Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250115 14:51:47.314239 416 test_util.cc:274] Using random seed: -786664942
2025-01-15T14:51:47Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-15T14:51:47Z Disabled control of system clock
I20250115 14:51:47.345809 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task__Z8ae/build/asan/bin/kudu
/tmp/dist-test-task__Z8ae/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35583
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250115 14:51:47.671051 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 14:51:47.671331 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 14:51:47.671418 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 14:51:47.679301 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250115 14:51:47.679421 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 14:51:47.679471 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250115 14:51:47.679524 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250115 14:51:47.688469 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4191
ASAN enabled
I20250115 14:51:47.690059 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 14:51:47.692173 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 14:51:47.702317 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:47.702870 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:47.703243 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:51:47.703800 424 server_base.cc:1034] running on GCE node
I20250115 14:51:47.704595 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 14:51:47.705991 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 14:51:47.707198 424 hybrid_clock.cc:648] HybridClock initialized: now 1736952707707108 us; error 68 us; skew 500 ppm
I20250115 14:51:47.707505 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 14:51:47.709816 424 webserver.cc:458] Webserver started at http://127.0.104.62:36213/ using document root <none> and password file <none>
I20250115 14:51:47.710530 424 fs_manager.cc:362] Metadata directory not provided
I20250115 14:51:47.710644 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 14:51:47.710927 424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 14:51:47.713459 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/data/instance:
uuid: "ef770dae727b42f5a9513b6f3d68b84b"
format_stamp: "Formatted at 2025-01-15 14:51:47 on dist-test-slave-pxr4"
I20250115 14:51:47.714167 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/wal/instance:
uuid: "ef770dae727b42f5a9513b6f3d68b84b"
format_stamp: "Formatted at 2025-01-15 14:51:47 on dist-test-slave-pxr4"
I20250115 14:51:47.719708 424 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250115 14:51:47.722950 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:47.724411 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 14:51:47.724558 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/data,/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/wal
uuid: "ef770dae727b42f5a9513b6f3d68b84b"
format_stamp: "Formatted at 2025-01-15 14:51:47 on dist-test-slave-pxr4"
I20250115 14:51:47.724738 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 14:51:47.742022 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 14:51:47.742863 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 14:51:47.743134 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 14:51:47.761540 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35583
I20250115 14:51:47.761584 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35583 every 8 connection(s)
I20250115 14:51:47.762905 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/data/info.pb
I20250115 14:51:47.766728 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:47.771262 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task__Z8ae/build/asan/bin/kudu as pid 424
I20250115 14:51:47.771530 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-0/wal/instance
I20250115 14:51:47.773051 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } has no permanent_uuid. Determining permanent_uuid...
I20250115 14:51:47.773977 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task__Z8ae/build/asan/bin/kudu
/tmp/dist-test-task__Z8ae/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:45653
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250115 14:51:47.793682 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } has no permanent_uuid. Determining permanent_uuid...
W20250115 14:51:47.794735 441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45653: connect: Connection refused (error 111)
W20250115 14:51:47.796165 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45653: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45653: connect: Connection refused (error 111)
I20250115 14:51:47.845582 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } attempt: 1
W20250115 14:51:47.847474 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45653: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45653: connect: Connection refused (error 111)
I20250115 14:51:47.915756 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } attempt: 2
W20250115 14:51:47.917629 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45653: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45653: connect: Connection refused (error 111)
I20250115 14:51:48.008976 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } attempt: 3
W20250115 14:51:48.010829 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45653: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45653: connect: Connection refused (error 111)
W20250115 14:51:48.106360 494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 14:51:48.106652 494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 14:51:48.106746 494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 14:51:48.114248 494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250115 14:51:48.114379 494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 14:51:48.114429 494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250115 14:51:48.114485 494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250115 14:51:48.124078 494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:45653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4191
ASAN enabled
I20250115 14:51:48.125625 494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 14:51:48.127244 494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 14:51:48.137965 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:48.138316 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:48.138752 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:51:48.139269 494 server_base.cc:1034] running on GCE node
I20250115 14:51:48.140040 494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 14:51:48.141503 494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 14:51:48.142725 494 hybrid_clock.cc:648] HybridClock initialized: now 1736952708142659 us; error 50 us; skew 500 ppm
I20250115 14:51:48.143046 494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 14:51:48.145238 494 webserver.cc:458] Webserver started at http://127.0.104.61:40195/ using document root <none> and password file <none>
I20250115 14:51:48.145977 494 fs_manager.cc:362] Metadata directory not provided
I20250115 14:51:48.146091 494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 14:51:48.146358 494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 14:51:48.148993 494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data/instance:
uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3"
format_stamp: "Formatted at 2025-01-15 14:51:48 on dist-test-slave-pxr4"
I20250115 14:51:48.150205 494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal/instance:
uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3"
format_stamp: "Formatted at 2025-01-15 14:51:48 on dist-test-slave-pxr4"
I20250115 14:51:48.154045 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } attempt: 4
I20250115 14:51:48.155246 494 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
W20250115 14:51:48.155704 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.61:45653: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45653: connect: Connection refused (error 111)
I20250115 14:51:48.158527 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:48.159785 494 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:48.159932 494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data,/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3"
format_stamp: "Formatted at 2025-01-15 14:51:48 on dist-test-slave-pxr4"
I20250115 14:51:48.160099 494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 14:51:48.187700 494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 14:51:48.188692 494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 14:51:48.188975 494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 14:51:48.207520 494 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:45653
I20250115 14:51:48.207547 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:45653 every 8 connection(s)
I20250115 14:51:48.208935 494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data/info.pb
I20250115 14:51:48.212584 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:48.218092 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } has no permanent_uuid. Determining permanent_uuid...
I20250115 14:51:48.219280 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task__Z8ae/build/asan/bin/kudu as pid 494
I20250115 14:51:48.219421 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal/instance
I20250115 14:51:48.220727 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task__Z8ae/build/asan/bin/kudu
/tmp/dist-test-task__Z8ae/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40227
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250115 14:51:48.233148 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } has no permanent_uuid. Determining permanent_uuid...
I20250115 14:51:48.242018 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } has no permanent_uuid. Determining permanent_uuid...
W20250115 14:51:48.243062 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111)
W20250115 14:51:48.244586 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:40227: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111)
I20250115 14:51:48.293893 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } attempt: 1
W20250115 14:51:48.295727 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:40227: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111)
I20250115 14:51:48.363946 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } attempt: 2
W20250115 14:51:48.365813 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:40227: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111)
I20250115 14:51:48.454916 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } attempt: 5
I20250115 14:51:48.457018 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } attempt: 3
W20250115 14:51:48.458739 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:40227: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111)
I20250115 14:51:48.462044 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } has no permanent_uuid. Determining permanent_uuid...
W20250115 14:51:48.463562 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:40227: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111)
I20250115 14:51:48.514787 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } attempt: 1
W20250115 14:51:48.517266 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:40227: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111)
W20250115 14:51:48.559855 564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 14:51:48.560173 564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 14:51:48.560258 564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 14:51:48.568099 564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250115 14:51:48.568243 564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 14:51:48.568331 564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250115 14:51:48.568404 564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250115 14:51:48.578307 564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4191
ASAN enabled
I20250115 14:51:48.579910 564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 14:51:48.581645 564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 14:51:48.585515 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } attempt: 2
W20250115 14:51:48.587518 490 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:40227: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111)
W20250115 14:51:48.592311 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:48.592831 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:48.593240 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:51:48.593767 564 server_base.cc:1034] running on GCE node
I20250115 14:51:48.594543 564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 14:51:48.595917 564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 14:51:48.597131 564 hybrid_clock.cc:648] HybridClock initialized: now 1736952708597058 us; error 46 us; skew 500 ppm
I20250115 14:51:48.597477 564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 14:51:48.599617 564 webserver.cc:458] Webserver started at http://127.0.104.60:37445/ using document root <none> and password file <none>
I20250115 14:51:48.600373 564 fs_manager.cc:362] Metadata directory not provided
I20250115 14:51:48.600482 564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 14:51:48.600766 564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 14:51:48.601943 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } attempt: 4
W20250115 14:51:48.603499 561 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.104.60:40227: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111)
I20250115 14:51:48.603718 564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/instance:
uuid: "1cc2e23c560f4096be7b14006bdfc525"
format_stamp: "Formatted at 2025-01-15 14:51:48 on dist-test-slave-pxr4"
I20250115 14:51:48.604465 564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal/instance:
uuid: "1cc2e23c560f4096be7b14006bdfc525"
format_stamp: "Formatted at 2025-01-15 14:51:48 on dist-test-slave-pxr4"
I20250115 14:51:48.609880 564 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250115 14:51:48.613058 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:48.614356 564 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250115 14:51:48.614558 564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data,/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
uuid: "1cc2e23c560f4096be7b14006bdfc525"
format_stamp: "Formatted at 2025-01-15 14:51:48 on dist-test-slave-pxr4"
I20250115 14:51:48.614763 564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 14:51:48.633284 564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 14:51:48.634130 564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 14:51:48.634397 564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 14:51:48.652645 564 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:40227
I20250115 14:51:48.652686 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:40227 every 8 connection(s)
I20250115 14:51:48.653976 564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/info.pb
I20250115 14:51:48.657344 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task__Z8ae/build/asan/bin/kudu as pid 564
I20250115 14:51:48.657514 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal/instance
I20250115 14:51:48.658236 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:48.665355 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } has no permanent_uuid. Determining permanent_uuid...
I20250115 14:51:48.677094 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } has no permanent_uuid. Determining permanent_uuid...
I20250115 14:51:48.684345 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } has no permanent_uuid. Determining permanent_uuid...
I20250115 14:51:48.693737 490 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } attempt: 3
I20250115 14:51:48.696432 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: Bootstrap starting.
I20250115 14:51:48.700273 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:48.701628 632 log.cc:826] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: Log is configured to *not* fsync() on all Append() calls
I20250115 14:51:48.704380 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: No bootstrap required, opened a new log
I20250115 14:51:48.704991 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Bootstrap starting.
I20250115 14:51:48.709254 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:48.710250 490 log.cc:826] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Log is configured to *not* fsync() on all Append() calls
I20250115 14:51:48.710958 632 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:48.711333 632 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:48.711454 632 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc2e23c560f4096be7b14006bdfc525, State: Initialized, Role: FOLLOWER
I20250115 14:51:48.712157 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:48.713249 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: No bootstrap required, opened a new log
I20250115 14:51:48.713089 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:48.713369 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:51:48.714116 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 14:51:48.720027 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:48.720430 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:48.720547 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef770dae727b42f5a9513b6f3d68b84b, State: Initialized, Role: FOLLOWER
I20250115 14:51:48.721236 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:48.722106 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:48.722424 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:51:48.723336 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: configured and running, proceeding with master startup.
W20250115 14:51:48.725250 653 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 14:51:48.725366 653 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 14:51:48.722581 564 master_runner.cc:186] Error getting master registration for 127.0.104.62:35583: OK, instance_id { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" instance_seqno: 1736952707755637 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250115 14:51:48.729768 564 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250115 14:51:48.738696 666 catalog_manager.cc:1559] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250115 14:51:48.738826 666 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250115 14:51:48.738097 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:45653: OK, instance_id { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" instance_seqno: 1736952708202138 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250115 14:51:48.746295 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250115 14:51:48.746269 444 tablet.cc:2367] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250115 14:51:48.886487 586 tablet.cc:2367] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 14:51:48.891197 654 raft_consensus.cc:491] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:48.891477 654 raft_consensus.cc:513] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:48.893232 654 leader_election.cc:290] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d97cf5f1fbb4bc6883711020e2fa3f3 (127.0.104.61:45653), 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227)
I20250115 14:51:48.894258 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef770dae727b42f5a9513b6f3d68b84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc2e23c560f4096be7b14006bdfc525" is_pre_election: true
I20250115 14:51:48.894850 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef770dae727b42f5a9513b6f3d68b84b in term 0.
I20250115 14:51:48.895543 441 leader_election.cc:304] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc2e23c560f4096be7b14006bdfc525, ef770dae727b42f5a9513b6f3d68b84b; no voters:
I20250115 14:51:48.896245 654 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:48.896440 654 raft_consensus.cc:491] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:48.896543 654 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:48.900220 654 raft_consensus.cc:513] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:48.900969 654 leader_election.cc:290] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [CANDIDATE]: Term 1 election: Requested vote from peers 7d97cf5f1fbb4bc6883711020e2fa3f3 (127.0.104.61:45653), 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227)
I20250115 14:51:48.901914 607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef770dae727b42f5a9513b6f3d68b84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cc2e23c560f4096be7b14006bdfc525"
I20250115 14:51:48.902136 607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:48.902704 561 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } attempt: 5
I20250115 14:51:48.906430 607 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef770dae727b42f5a9513b6f3d68b84b in term 1.
I20250115 14:51:48.916498 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: Bootstrap starting.
I20250115 14:51:48.920643 441 leader_election.cc:304] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cc2e23c560f4096be7b14006bdfc525, ef770dae727b42f5a9513b6f3d68b84b; no voters:
I20250115 14:51:48.921268 654 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:48.923142 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:48.924544 561 log.cc:826] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: Log is configured to *not* fsync() on all Append() calls
I20250115 14:51:48.924823 654 raft_consensus.cc:695] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [term 1 LEADER]: Becoming Leader. State: Replica: ef770dae727b42f5a9513b6f3d68b84b, State: Running, Role: LEADER
I20250115 14:51:48.925351 654 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:48.929736 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef770dae727b42f5a9513b6f3d68b84b. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:48.930075 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: This master's current role is: LEADER
I20250115 14:51:48.932251 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: No bootstrap required, opened a new log
I20250115 14:51:48.938380 671 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250115 14:51:48.941375 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:48.941846 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:48.942014 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d97cf5f1fbb4bc6883711020e2fa3f3, State: Initialized, Role: FOLLOWER
I20250115 14:51:48.942667 671 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250115 14:51:48.942977 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:48.949971 672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:48.952348 535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef770dae727b42f5a9513b6f3d68b84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3"
I20250115 14:51:48.952867 535 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:48.956794 535 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef770dae727b42f5a9513b6f3d68b84b in term 1.
I20250115 14:51:48.952080 536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef770dae727b42f5a9513b6f3d68b84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" is_pre_election: true
I20250115 14:51:48.957363 536 raft_consensus.cc:2371] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ef770dae727b42f5a9513b6f3d68b84b in term 1. Re-sending same reply.
I20250115 14:51:48.952564 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 14:51:48.959899 672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:51:48.964892 607 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [term 1 FOLLOWER]: Refusing update from remote peer ef770dae727b42f5a9513b6f3d68b84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:48.965938 654 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:48.966516 535 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Refusing update from remote peer ef770dae727b42f5a9513b6f3d68b84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:48.970881 654 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:48.991065 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef770dae727b42f5a9513b6f3d68b84b. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:48.991402 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:51:48.995741 678 mvcc.cc:204] Tried to move back new op lower bound from 7114558295904444416 to 7114558295763329024. Current Snapshot: MvccSnapshot[applied={T|T < 7114558295904444416}]
I20250115 14:51:49.002616 672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef770dae727b42f5a9513b6f3d68b84b. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:49.002959 672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:51:49.008714 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:49.008977 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:51:49.020648 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:49.020622 672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:49.020890 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: This master's current role is: LEADER
I20250115 14:51:49.020895 672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:51:49.021121 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:49.021306 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: This master's current role is: LEADER
I20250115 14:51:49.023150 671 catalog_manager.cc:1348] Generated new cluster ID: 838a9b5bdaf340e2b93e4cbb7e53a2a4
I20250115 14:51:49.023267 671 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250115 14:51:49.055351 671 catalog_manager.cc:1371] Generated new certificate authority record
I20250115 14:51:49.057399 671 catalog_manager.cc:1505] Loading token signing keys...
I20250115 14:51:49.074584 671 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Generated new TSK 0
I20250115 14:51:49.075691 671 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250115 14:51:49.090595 691 catalog_manager.cc:1260] Loaded cluster ID: 838a9b5bdaf340e2b93e4cbb7e53a2a4
I20250115 14:51:49.090808 691 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: loading cluster ID for follower catalog manager: success
I20250115 14:51:49.094697 691 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: acquiring CA information for follower catalog manager: success
I20250115 14:51:49.097651 691 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 14:51:49.111269 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task__Z8ae/build/asan/bin/kudu
/tmp/dist-test-task__Z8ae/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--builtin_ntp_servers=127.0.104.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250115 14:51:49.514600 693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 14:51:49.515067 693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 14:51:49.515236 693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 14:51:49.528344 693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 14:51:49.528637 693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250115 14:51:49.543267 693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4191
ASAN enabled
I20250115 14:51:49.545279 693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 14:51:49.547695 693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 14:51:49.559345 698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:49.559850 701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:49.560807 699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:51:49.560881 693 server_base.cc:1034] running on GCE node
I20250115 14:51:49.561564 693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 14:51:49.562487 693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 14:51:49.563719 693 hybrid_clock.cc:648] HybridClock initialized: now 1736952709563638 us; error 49 us; skew 500 ppm
I20250115 14:51:49.564024 693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 14:51:49.566288 693 webserver.cc:458] Webserver started at http://127.0.104.1:32779/ using document root <none> and password file <none>
I20250115 14:51:49.567006 693 fs_manager.cc:362] Metadata directory not provided
I20250115 14:51:49.567118 693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 14:51:49.567390 693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 14:51:49.569995 693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/data/instance:
uuid: "d2d7ecece2ab49e1be7079216517d13b"
format_stamp: "Formatted at 2025-01-15 14:51:49 on dist-test-slave-pxr4"
I20250115 14:51:49.570686 693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/instance:
uuid: "d2d7ecece2ab49e1be7079216517d13b"
format_stamp: "Formatted at 2025-01-15 14:51:49 on dist-test-slave-pxr4"
I20250115 14:51:49.575731 693 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250115 14:51:49.577378 707 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef770dae727b42f5a9513b6f3d68b84b)
I20250115 14:51:49.577540 707 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:49.578958 707 leader_election.cc:290] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef770dae727b42f5a9513b6f3d68b84b (127.0.104.62:35583), 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227)
I20250115 14:51:49.579268 708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:49.579347 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1cc2e23c560f4096be7b14006bdfc525" is_pre_election: true
I20250115 14:51:49.579543 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef770dae727b42f5a9513b6f3d68b84b" is_pre_election: true
I20250115 14:51:49.580385 511 leader_election.cc:304] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d97cf5f1fbb4bc6883711020e2fa3f3; no voters: 1cc2e23c560f4096be7b14006bdfc525, ef770dae727b42f5a9513b6f3d68b84b
I20250115 14:51:49.580708 707 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 14:51:49.581203 693 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250115 14:51:49.581346 693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/data,/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal
uuid: "d2d7ecece2ab49e1be7079216517d13b"
format_stamp: "Formatted at 2025-01-15 14:51:49 on dist-test-slave-pxr4"
I20250115 14:51:49.581498 693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 14:51:49.613170 693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 14:51:49.614006 693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 14:51:49.614274 693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 14:51:49.615198 693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 14:51:49.616680 693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 14:51:49.616760 693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:49.616879 693 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 14:51:49.616930 693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:49.647778 693 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:34127
I20250115 14:51:49.647842 820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:34127 every 8 connection(s)
I20250115 14:51:49.649094 693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/data/info.pb
I20250115 14:51:49.651825 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task__Z8ae/build/asan/bin/kudu as pid 693
I20250115 14:51:49.652017 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/instance
I20250115 14:51:49.657285 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task__Z8ae/build/asan/bin/kudu
/tmp/dist-test-task__Z8ae/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--builtin_ntp_servers=127.0.104.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250115 14:51:49.673978 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:51:49.674343 823 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:49.675366 823 heartbeater.cc:509] Master 127.0.104.61:45653 requested a full tablet report, sending...
I20250115 14:51:49.677703 526 ts_manager.cc:194] Registered new tserver with Master: d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127)
I20250115 14:51:49.681597 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:49.681725 822 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:49.682209 821 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35583
I20250115 14:51:49.682226 822 heartbeater.cc:509] Master 127.0.104.60:40227 requested a full tablet report, sending...
I20250115 14:51:49.682351 821 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:49.682792 821 heartbeater.cc:509] Master 127.0.104.62:35583 requested a full tablet report, sending...
I20250115 14:51:49.683822 597 ts_manager.cc:194] Registered new tserver with Master: d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127)
I20250115 14:51:49.684216 455 ts_manager.cc:194] Registered new tserver with Master: d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127)
I20250115 14:51:49.686234 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:59603
I20250115 14:51:49.727681 653 catalog_manager.cc:1260] Loaded cluster ID: 838a9b5bdaf340e2b93e4cbb7e53a2a4
I20250115 14:51:49.727794 653 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: loading cluster ID for follower catalog manager: success
I20250115 14:51:49.731267 653 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: acquiring CA information for follower catalog manager: success
I20250115 14:51:49.733347 653 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250115 14:51:49.993212 831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 14:51:49.993474 831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 14:51:49.993566 831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 14:51:50.001417 831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 14:51:50.001621 831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250115 14:51:50.010866 831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4191
ASAN enabled
I20250115 14:51:50.012377 831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 14:51:50.014046 831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 14:51:50.025730 836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:50.026265 837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:50.026683 839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:51:50.027228 831 server_base.cc:1034] running on GCE node
I20250115 14:51:50.027786 831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 14:51:50.028687 831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 14:51:50.029886 831 hybrid_clock.cc:648] HybridClock initialized: now 1736952710029820 us; error 42 us; skew 500 ppm
I20250115 14:51:50.030186 831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 14:51:50.032790 831 webserver.cc:458] Webserver started at http://127.0.104.2:33223/ using document root <none> and password file <none>
I20250115 14:51:50.033474 831 fs_manager.cc:362] Metadata directory not provided
I20250115 14:51:50.033581 831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 14:51:50.033844 831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 14:51:50.036314 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/data/instance:
uuid: "73c86158cc2641f182ea50a06634592d"
format_stamp: "Formatted at 2025-01-15 14:51:50 on dist-test-slave-pxr4"
I20250115 14:51:50.037104 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/instance:
uuid: "73c86158cc2641f182ea50a06634592d"
format_stamp: "Formatted at 2025-01-15 14:51:50 on dist-test-slave-pxr4"
I20250115 14:51:50.042052 831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250115 14:51:50.045141 845 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:50.046314 831 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250115 14:51:50.046500 831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/data,/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal
uuid: "73c86158cc2641f182ea50a06634592d"
format_stamp: "Formatted at 2025-01-15 14:51:50 on dist-test-slave-pxr4"
I20250115 14:51:50.046670 831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 14:51:50.071004 831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 14:51:50.071815 831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 14:51:50.072105 831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 14:51:50.073050 831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 14:51:50.074529 831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 14:51:50.074606 831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:50.074718 831 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 14:51:50.074772 831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:50.106793 831 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:44187
I20250115 14:51:50.106827 957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:44187 every 8 connection(s)
I20250115 14:51:50.108000 831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/data/info.pb
I20250115 14:51:50.108430 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task__Z8ae/build/asan/bin/kudu as pid 831
I20250115 14:51:50.108640 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/instance
I20250115 14:51:50.113337 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task__Z8ae/build/asan/bin/kudu
/tmp/dist-test-task__Z8ae/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--builtin_ntp_servers=127.0.104.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250115 14:51:50.126600 958 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35583
I20250115 14:51:50.126947 958 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:50.127985 958 heartbeater.cc:509] Master 127.0.104.62:35583 requested a full tablet report, sending...
I20250115 14:51:50.129637 455 ts_manager.cc:194] Registered new tserver with Master: 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:50.130643 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:50927
I20250115 14:51:50.133810 966 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef770dae727b42f5a9513b6f3d68b84b)
I20250115 14:51:50.134042 966 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:50.134662 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:51:50.134791 960 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:50.135152 960 heartbeater.cc:509] Master 127.0.104.61:45653 requested a full tablet report, sending...
I20250115 14:51:50.135251 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ef770dae727b42f5a9513b6f3d68b84b" is_pre_election: true
I20250115 14:51:50.135600 606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1cc2e23c560f4096be7b14006bdfc525" is_pre_election: true
I20250115 14:51:50.136219 959 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:50.136229 526 ts_manager.cc:194] Registered new tserver with Master: 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:50.136404 959 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:50.136152 512 leader_election.cc:304] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d97cf5f1fbb4bc6883711020e2fa3f3; no voters: 1cc2e23c560f4096be7b14006bdfc525, ef770dae727b42f5a9513b6f3d68b84b
I20250115 14:51:50.136662 959 heartbeater.cc:509] Master 127.0.104.60:40227 requested a full tablet report, sending...
I20250115 14:51:50.136157 966 leader_election.cc:290] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef770dae727b42f5a9513b6f3d68b84b (127.0.104.62:35583), 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227)
I20250115 14:51:50.137231 966 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 14:51:50.137663 597 ts_manager.cc:194] Registered new tserver with Master: 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:50.188817 821 heartbeater.cc:501] Master 127.0.104.62:35583 was elected leader, sending a full tablet report...
W20250115 14:51:50.456219 965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 14:51:50.456512 965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 14:51:50.456646 965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250115 14:51:50.464109 965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 14:51:50.464351 965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250115 14:51:50.473558 965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4191
ASAN enabled
I20250115 14:51:50.475064 965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 14:51:50.476711 965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250115 14:51:50.487772 975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:50.488155 972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:50.488595 973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:51:50.489213 965 server_base.cc:1034] running on GCE node
I20250115 14:51:50.489795 965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 14:51:50.490604 965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 14:51:50.491806 965 hybrid_clock.cc:648] HybridClock initialized: now 1736952710491737 us; error 50 us; skew 500 ppm
I20250115 14:51:50.492059 965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 14:51:50.494118 965 webserver.cc:458] Webserver started at http://127.0.104.3:42343/ using document root <none> and password file <none>
I20250115 14:51:50.494786 965 fs_manager.cc:362] Metadata directory not provided
I20250115 14:51:50.494894 965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 14:51:50.495157 965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250115 14:51:50.497571 965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/data/instance:
uuid: "c238c037a1c74725a001e2b4a1ff4304"
format_stamp: "Formatted at 2025-01-15 14:51:50 on dist-test-slave-pxr4"
I20250115 14:51:50.498245 965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/instance:
uuid: "c238c037a1c74725a001e2b4a1ff4304"
format_stamp: "Formatted at 2025-01-15 14:51:50 on dist-test-slave-pxr4"
I20250115 14:51:50.503464 965 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20250115 14:51:50.506511 981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:50.507613 965 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:50.507767 965 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/data,/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal
uuid: "c238c037a1c74725a001e2b4a1ff4304"
format_stamp: "Formatted at 2025-01-15 14:51:50 on dist-test-slave-pxr4"
I20250115 14:51:50.507916 965 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 14:51:50.543948 965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 14:51:50.544785 965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 14:51:50.545043 965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 14:51:50.545903 965 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250115 14:51:50.547317 965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250115 14:51:50.547391 965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:50.547505 965 ts_tablet_manager.cc:610] Registered 0 tablets
I20250115 14:51:50.547555 965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:50.578655 965 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36351
I20250115 14:51:50.578716 1093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36351 every 8 connection(s)
I20250115 14:51:50.579878 965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/data/info.pb
I20250115 14:51:50.580404 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task__Z8ae/build/asan/bin/kudu as pid 965
I20250115 14:51:50.580559 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/instance
I20250115 14:51:50.597205 1094 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35583
I20250115 14:51:50.597493 1094 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:50.597724 1095 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:50.597745 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:51:50.597920 1095 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:50.597919 1096 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:50.598810 1094 heartbeater.cc:509] Master 127.0.104.62:35583 requested a full tablet report, sending...
I20250115 14:51:50.599086 1095 heartbeater.cc:509] Master 127.0.104.60:40227 requested a full tablet report, sending...
I20250115 14:51:50.599109 1096 heartbeater.cc:509] Master 127.0.104.61:45653 requested a full tablet report, sending...
I20250115 14:51:50.600147 455 ts_manager.cc:194] Registered new tserver with Master: c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:50.600315 597 ts_manager.cc:194] Registered new tserver with Master: c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:50.600376 526 ts_manager.cc:194] Registered new tserver with Master: c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:50.601086 455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:55879
I20250115 14:51:50.606173 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250115 14:51:50.632164 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6778ac942eae482e9ca0d7b8993b4233"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:50.633049 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4a983082b9b74279b479488150f64d8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:50.633123 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5dc669c911b341b7b3f63a5af4487c6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:50.633734 958 heartbeater.cc:501] Master 127.0.104.62:35583 was elected leader, sending a full tablet report...
W20250115 14:51:50.633877 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6778ac942eae482e9ca0d7b8993b4233 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:50.634076 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dc669c911b341b7b3f63a5af4487c6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.635180 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6110d5011f0744fcaa9054846fad2c3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.635677 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6110d5011f0744fcaa9054846fad2c3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.637835 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5e1de22b982e4acca9f681957c100519"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.638444 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e1de22b982e4acca9f681957c100519 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.639320 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_10be1d5f636948889ab51d4a914d239b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.639879 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10be1d5f636948889ab51d4a914d239b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:50.641724 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a983082b9b74279b479488150f64d8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.646054 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2b49018445c943b4ad081ae507a278eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:50.646133 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_96e6f9300a064e6b98d7f41b613f26e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.646736 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b49018445c943b4ad081ae507a278eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:50.646857 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96e6f9300a064e6b98d7f41b613f26e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.693454 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7e0377b7d29440f18ad8eac96ccf8766"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.694628 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e0377b7d29440f18ad8eac96ccf8766 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.705024 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task__Z8ae/build/asan/bin/kudu pid 564
I20250115 14:51:50.706107 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20250115 14:51:50.702010 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_692ffd08893a41cfa029ad2c470f1735"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:50.709342 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5dc669c911b341b7b3f63a5af4487c6b" } new_table_name: "default.Table_536058ab838e47e89d30797e48106518" modify_external_catalogs: true
W20250115 14:51:50.710960 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_692ffd08893a41cfa029ad2c470f1735 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.713599 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5e1de22b982e4acca9f681957c100519" } new_table_name: "default.Table_b1b40a84f1874354bf55f4442aa17d8b" modify_external_catalogs: true
I20250115 14:51:50.713621 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6110d5011f0744fcaa9054846fad2c3b" } new_table_name: "default.Table_84ae57866f144c9ba1375c09d773bf4f" modify_external_catalogs: true
I20250115 14:51:50.728206 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_10be1d5f636948889ab51d4a914d239b" } new_table_name: "default.Table_4d560275be034a7cb7e0f17eec823c1e" modify_external_catalogs: true
W20250115 14:51:50.732578 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 649504c31def4f3f9c30879033b58fbe consensus state has no leader
W20250115 14:51:50.762668 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 77eb5ea1789d47eb8a9379923089f687 has no consensus state
I20250115 14:51:50.762854 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_96e6f9300a064e6b98d7f41b613f26e6" } modify_external_catalogs: true
I20250115 14:51:50.763118 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_96e6f9300a064e6b98d7f41b613f26e6" } modify_external_catalogs: true
I20250115 14:51:50.764611 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6778ac942eae482e9ca0d7b8993b4233" } modify_external_catalogs: true
I20250115 14:51:50.764794 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6778ac942eae482e9ca0d7b8993b4233" } modify_external_catalogs: true
I20250115 14:51:50.765125 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_692ffd08893a41cfa029ad2c470f1735" } new_table_name: "default.Table_9460bdb384754c468295752de05d4457" modify_external_catalogs: true
I20250115 14:51:50.766285 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7e0377b7d29440f18ad8eac96ccf8766" } modify_external_catalogs: true
I20250115 14:51:50.766443 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7e0377b7d29440f18ad8eac96ccf8766" } modify_external_catalogs: true
I20250115 14:51:50.768692 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4a983082b9b74279b479488150f64d8e" } modify_external_catalogs: true
I20250115 14:51:50.768865 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4a983082b9b74279b479488150f64d8e" } modify_external_catalogs: true
W20250115 14:51:50.764854 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1328f038ad3423a8f0edcb13e0f555a has no consensus state
I20250115 14:51:50.769402 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2b49018445c943b4ad081ae507a278eb" } modify_external_catalogs: true
I20250115 14:51:50.769546 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2b49018445c943b4ad081ae507a278eb" } modify_external_catalogs: true
W20250115 14:51:50.773312 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7bdb617ac1c042d282873f2c01bf6162 has no consensus state
I20250115 14:51:50.776319 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 649504c31def4f3f9c30879033b58fbe (DEFAULT_TABLE table=default.Table_5dc669c911b341b7b3f63a5af4487c6b [id=f6f31e321abb451c877540a31b2fb976]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.776325 756 tablet_service.cc:1467] Processing CreateTablet for tablet 649504c31def4f3f9c30879033b58fbe (DEFAULT_TABLE table=default.Table_5dc669c911b341b7b3f63a5af4487c6b [id=f6f31e321abb451c877540a31b2fb976]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.777784 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 649504c31def4f3f9c30879033b58fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.777798 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 649504c31def4f3f9c30879033b58fbe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:50.780779 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 649504c31def4f3f9c30879033b58fbe (table default.Table_536058ab838e47e89d30797e48106518 [id=f6f31e321abb451c877540a31b2fb976], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 649504c31def4f3f9c30879033b58fbe consensus state has no leader
W20250115 14:51:50.787621 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7249642b69844b6d95fd9c8419e122d6 has no consensus state
I20250115 14:51:50.787756 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 4cff64b58e8b4f5ab9aafa1693338c8e
I20250115 14:51:50.788003 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 9393c58a0108492e88f22e48c1b222a9
I20250115 14:51:50.788383 448 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 315df66a8c60461cab6d76d999ebd872
I20250115 14:51:50.789081 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b76d971eccdb467b8923a28ac63730a4
I20250115 14:51:50.804834 755 tablet_service.cc:1467] Processing CreateTablet for tablet 7249642b69844b6d95fd9c8419e122d6 (DEFAULT_TABLE table=default.Table_9460bdb384754c468295752de05d4457 [id=1736b22b006f4ad686507bf204afc585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.805814 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 77eb5ea1789d47eb8a9379923089f687 (DEFAULT_TABLE table=default.Table_84ae57866f144c9ba1375c09d773bf4f [id=8767564aae4d4fd198b9c9ea66abe032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.805969 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7249642b69844b6d95fd9c8419e122d6 (DEFAULT_TABLE table=default.Table_9460bdb384754c468295752de05d4457 [id=1736b22b006f4ad686507bf204afc585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.806425 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77eb5ea1789d47eb8a9379923089f687. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.806792 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 17745a9f088946dba78e0630f7400a2e (DEFAULT_TABLE table=default.Table_4a983082b9b74279b479488150f64d8e [id=6f3d691f400a4cccaf708d491a237b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.806886 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdb617ac1c042d282873f2c01bf6162 (DEFAULT_TABLE table=default.Table_4d560275be034a7cb7e0f17eec823c1e [id=ff5b7f4d675f496eaf7dd60b0cca7435]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.807307 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17745a9f088946dba78e0630f7400a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.807940 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bdb617ac1c042d282873f2c01bf6162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.806427 1026 tablet_service.cc:1467] Processing CreateTablet for tablet f1328f038ad3423a8f0edcb13e0f555a (DEFAULT_TABLE table=default.Table_b1b40a84f1874354bf55f4442aa17d8b [id=d7ba4cda40964e1d839efe9e0f1615c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.809330 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1328f038ad3423a8f0edcb13e0f555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.810914 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7249642b69844b6d95fd9c8419e122d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.811069 1162 tablet_bootstrap.cc:492] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:50.814826 754 tablet_service.cc:1467] Processing CreateTablet for tablet 77eb5ea1789d47eb8a9379923089f687 (DEFAULT_TABLE table=default.Table_84ae57866f144c9ba1375c09d773bf4f [id=8767564aae4d4fd198b9c9ea66abe032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.815721 753 tablet_service.cc:1467] Processing CreateTablet for tablet f1328f038ad3423a8f0edcb13e0f555a (DEFAULT_TABLE table=default.Table_b1b40a84f1874354bf55f4442aa17d8b [id=d7ba4cda40964e1d839efe9e0f1615c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.815968 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdb617ac1c042d282873f2c01bf6162 (DEFAULT_TABLE table=default.Table_4d560275be034a7cb7e0f17eec823c1e [id=ff5b7f4d675f496eaf7dd60b0cca7435]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.816908 751 tablet_service.cc:1467] Processing CreateTablet for tablet 17745a9f088946dba78e0630f7400a2e (DEFAULT_TABLE table=default.Table_4a983082b9b74279b479488150f64d8e [id=6f3d691f400a4cccaf708d491a237b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.817274 1162 tablet_bootstrap.cc:654] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:50.817989 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7249642b69844b6d95fd9c8419e122d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.818792 1162 log.cc:826] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: Log is configured to *not* fsync() on all Append() calls
I20250115 14:51:50.824071 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17745a9f088946dba78e0630f7400a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.824664 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bdb617ac1c042d282873f2c01bf6162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.828452 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1328f038ad3423a8f0edcb13e0f555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.833324 892 tablet_service.cc:1467] Processing CreateTablet for tablet 7249642b69844b6d95fd9c8419e122d6 (DEFAULT_TABLE table=default.Table_9460bdb384754c468295752de05d4457 [id=1736b22b006f4ad686507bf204afc585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.834352 891 tablet_service.cc:1467] Processing CreateTablet for tablet 77eb5ea1789d47eb8a9379923089f687 (DEFAULT_TABLE table=default.Table_84ae57866f144c9ba1375c09d773bf4f [id=8767564aae4d4fd198b9c9ea66abe032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.834879 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7249642b69844b6d95fd9c8419e122d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.834892 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 17745a9f088946dba78e0630f7400a2e
I20250115 14:51:50.834965 893 tablet_service.cc:1467] Processing CreateTablet for tablet 649504c31def4f3f9c30879033b58fbe (DEFAULT_TABLE table=default.Table_5dc669c911b341b7b3f63a5af4487c6b [id=f6f31e321abb451c877540a31b2fb976]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.835397 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 649504c31def4f3f9c30879033b58fbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.836354 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 17745a9f088946dba78e0630f7400a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:50.836432 890 tablet_service.cc:1467] Processing CreateTablet for tablet f1328f038ad3423a8f0edcb13e0f555a (DEFAULT_TABLE table=default.Table_b1b40a84f1874354bf55f4442aa17d8b [id=d7ba4cda40964e1d839efe9e0f1615c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.836735 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 17745a9f088946dba78e0630f7400a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:50.837841 888 tablet_service.cc:1467] Processing CreateTablet for tablet 17745a9f088946dba78e0630f7400a2e (DEFAULT_TABLE table=default.Table_4a983082b9b74279b479488150f64d8e [id=6f3d691f400a4cccaf708d491a237b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.836570 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 17745a9f088946dba78e0630f7400a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:50.834882 889 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdb617ac1c042d282873f2c01bf6162 (DEFAULT_TABLE table=default.Table_4d560275be034a7cb7e0f17eec823c1e [id=ff5b7f4d675f496eaf7dd60b0cca7435]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.839622 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77eb5ea1789d47eb8a9379923089f687. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:50.841567 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 17745a9f088946dba78e0630f7400a2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17745a9f088946dba78e0630f7400a2e already in progress: creating tablet
I20250115 14:51:50.844806 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1328f038ad3423a8f0edcb13e0f555a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.848464 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bdb617ac1c042d282873f2c01bf6162. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.852447 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17745a9f088946dba78e0630f7400a2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.852703 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77eb5ea1789d47eb8a9379923089f687. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.863296 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 17745a9f088946dba78e0630f7400a2e from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:50.863974 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 17745a9f088946dba78e0630f7400a2e from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:50.864826 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 77eb5ea1789d47eb8a9379923089f687 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:50.864953 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 77eb5ea1789d47eb8a9379923089f687 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:50.871888 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 77eb5ea1789d47eb8a9379923089f687
I20250115 14:51:50.872442 449 catalog_manager.cc:6596] ReplaceTablet: tablet 77eb5ea1789d47eb8a9379923089f687 deleted and replaced by tablet 510ac6d6ee3e488e8b20f3e39a44ab9a
I20250115 14:51:50.875017 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 77eb5ea1789d47eb8a9379923089f687 with delete_type TABLET_DATA_DELETED (replaced by tablet 510ac6d6ee3e488e8b20f3e39a44ab9a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:50.876788 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 77eb5ea1789d47eb8a9379923089f687 with delete_type TABLET_DATA_DELETED (replaced by tablet 510ac6d6ee3e488e8b20f3e39a44ab9a) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:50.879827 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a (DEFAULT_TABLE table=default.Table_84ae57866f144c9ba1375c09d773bf4f [id=8767564aae4d4fd198b9c9ea66abe032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.880260 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.883260 893 tablet_service.cc:1467] Processing CreateTablet for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a (DEFAULT_TABLE table=default.Table_84ae57866f144c9ba1375c09d773bf4f [id=8767564aae4d4fd198b9c9ea66abe032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.883626 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.886740 1164 tablet_bootstrap.cc:492] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:50.886778 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 77eb5ea1789d47eb8a9379923089f687 with delete_type TABLET_DATA_DELETED (replaced by tablet 510ac6d6ee3e488e8b20f3e39a44ab9a) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:50.887490 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 77eb5ea1789d47eb8a9379923089f687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77eb5ea1789d47eb8a9379923089f687 already in progress: creating tablet
I20250115 14:51:50.887470 754 tablet_service.cc:1467] Processing CreateTablet for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a (DEFAULT_TABLE table=default.Table_84ae57866f144c9ba1375c09d773bf4f [id=8767564aae4d4fd198b9c9ea66abe032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.887980 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.892099 1164 tablet_bootstrap.cc:654] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:50.894724 1162 tablet_bootstrap.cc:492] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:50.895025 1162 ts_tablet_manager.cc:1397] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.004s
W20250115 14:51:50.896049 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 17745a9f088946dba78e0630f7400a2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17745a9f088946dba78e0630f7400a2e already in progress: creating tablet
I20250115 14:51:50.899210 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f04c4734007f45f2b1b6dbe3cfeec857"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.899422 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 77eb5ea1789d47eb8a9379923089f687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77eb5ea1789d47eb8a9379923089f687 already in progress: creating tablet
W20250115 14:51:50.899816 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f04c4734007f45f2b1b6dbe3cfeec857 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.900929 1164 log.cc:826] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: Log is configured to *not* fsync() on all Append() calls
W20250115 14:51:50.902379 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 17745a9f088946dba78e0630f7400a2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17745a9f088946dba78e0630f7400a2e already in progress: creating tablet
I20250115 14:51:50.904881 1162 raft_consensus.cc:357] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:50.905906 1168 tablet_bootstrap.cc:492] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:50.906847 1162 raft_consensus.cc:383] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:50.906963 1162 raft_consensus.cc:738] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:50.907721 1162 consensus_queue.cc:260] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:50.909971 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 77eb5ea1789d47eb8a9379923089f687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77eb5ea1789d47eb8a9379923089f687 already in progress: creating tablet
I20250115 14:51:50.910820 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_07c97d6a492c476291f8fa126fabba8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.911353 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07c97d6a492c476291f8fa126fabba8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.912586 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 7bdb617ac1c042d282873f2c01bf6162 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:50.914777 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.915225 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.915735 1168 tablet_bootstrap.cc:654] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:50.916399 888 tablet_service.cc:1467] Processing CreateTablet for tablet 101087eee11441f299d5b75e5c80b66b (DEFAULT_TABLE table=default.Table_f04c4734007f45f2b1b6dbe3cfeec857 [id=05c9f91a89fb43c8918c6cfe74360e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.916798 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101087eee11441f299d5b75e5c80b66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.916960 1168 log.cc:826] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: Log is configured to *not* fsync() on all Append() calls
I20250115 14:51:50.919657 1164 tablet_bootstrap.cc:492] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:50.919907 1164 ts_tablet_manager.cc:1397] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.004s
I20250115 14:51:50.921175 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 101087eee11441f299d5b75e5c80b66b (DEFAULT_TABLE table=default.Table_f04c4734007f45f2b1b6dbe3cfeec857 [id=05c9f91a89fb43c8918c6cfe74360e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.921550 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101087eee11441f299d5b75e5c80b66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.921869 754 tablet_service.cc:1467] Processing CreateTablet for tablet 101087eee11441f299d5b75e5c80b66b (DEFAULT_TABLE table=default.Table_f04c4734007f45f2b1b6dbe3cfeec857 [id=05c9f91a89fb43c8918c6cfe74360e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.922253 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 101087eee11441f299d5b75e5c80b66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.925737 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fb13e7bf7ead4609a5d43f200ab36cad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.926235 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb13e7bf7ead4609a5d43f200ab36cad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.926411 1168 tablet_bootstrap.cc:492] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:50.926645 1168 ts_tablet_manager.cc:1397] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20250115 14:51:50.927338 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0bf040cafd794a0d92935f720d3e8b09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.927764 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bf040cafd794a0d92935f720d3e8b09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.929461 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5efe284aa9074f8b841d0ccde93e7c20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:50.929559 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5d2e74e77f87456396aed66d4d21f840"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.929922 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5efe284aa9074f8b841d0ccde93e7c20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:50.929991 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d2e74e77f87456396aed66d4d21f840 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.930799 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0181146ed3ba475d86d8e73223c9624b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.931195 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0181146ed3ba475d86d8e73223c9624b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.935422 1094 heartbeater.cc:501] Master 127.0.104.62:35583 was elected leader, sending a full tablet report...
I20250115 14:51:50.935751 1164 raft_consensus.cc:357] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:50.936115 1162 ts_tablet_manager.cc:1428] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.041s user 0.012s sys 0.009s
I20250115 14:51:50.936156 1164 raft_consensus.cc:383] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:50.936257 1164 raft_consensus.cc:738] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:50.937404 1162 tablet_bootstrap.cc:492] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:50.939273 1162 tablet_bootstrap.cc:654] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:50.942966 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task__Z8ae/build/asan/bin/kudu with pid 564
I20250115 14:51:50.946061 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7bdb617ac1c042d282873f2c01bf6162
I20250115 14:51:50.946529 448 catalog_manager.cc:6596] ReplaceTablet: tablet 7bdb617ac1c042d282873f2c01bf6162 deleted and replaced by tablet 59f25f2f228143c98f6cd6ccb0cf99d7
I20250115 14:51:50.947993 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb617ac1c042d282873f2c01bf6162 with delete_type TABLET_DATA_DELETED (replaced by tablet 59f25f2f228143c98f6cd6ccb0cf99d7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:50.948048 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 649504c31def4f3f9c30879033b58fbe (table default.Table_536058ab838e47e89d30797e48106518 [id=f6f31e321abb451c877540a31b2fb976]). Expected version 1 got 0
I20250115 14:51:50.948242 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a (table default.Table_84ae57866f144c9ba1375c09d773bf4f [id=8767564aae4d4fd198b9c9ea66abe032]). Expected version 1 got 0
I20250115 14:51:50.948469 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7249642b69844b6d95fd9c8419e122d6 (table default.Table_9460bdb384754c468295752de05d4457 [id=1736b22b006f4ad686507bf204afc585]). Expected version 1 got 0
I20250115 14:51:50.948637 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet f1328f038ad3423a8f0edcb13e0f555a (table default.Table_b1b40a84f1874354bf55f4442aa17d8b [id=d7ba4cda40964e1d839efe9e0f1615c0]). Expected version 1 got 0
I20250115 14:51:50.949260 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb617ac1c042d282873f2c01bf6162 with delete_type TABLET_DATA_DELETED (replaced by tablet 59f25f2f228143c98f6cd6ccb0cf99d7) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:50.950171 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a (table default.Table_84ae57866f144c9ba1375c09d773bf4f [id=8767564aae4d4fd198b9c9ea66abe032], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 510ac6d6ee3e488e8b20f3e39a44ab9a consensus state has no leader
W20250115 14:51:50.952947 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7bdb617ac1c042d282873f2c01bf6162 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bdb617ac1c042d282873f2c01bf6162 already in progress: creating tablet
W20250115 14:51:50.954751 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7bdb617ac1c042d282873f2c01bf6162 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bdb617ac1c042d282873f2c01bf6162 already in progress: creating tablet
I20250115 14:51:50.955348 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 17745a9f088946dba78e0630f7400a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:50.955348 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 77eb5ea1789d47eb8a9379923089f687 with delete_type TABLET_DATA_DELETED (replaced by tablet 510ac6d6ee3e488e8b20f3e39a44ab9a) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:50.956719 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 77eb5ea1789d47eb8a9379923089f687 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77eb5ea1789d47eb8a9379923089f687 already in progress: creating tablet
W20250115 14:51:50.956946 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 17745a9f088946dba78e0630f7400a2e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17745a9f088946dba78e0630f7400a2e already in progress: creating tablet
I20250115 14:51:50.958422 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb617ac1c042d282873f2c01bf6162 with delete_type TABLET_DATA_DELETED (replaced by tablet 59f25f2f228143c98f6cd6ccb0cf99d7) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:50.959491 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7bdb617ac1c042d282873f2c01bf6162 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bdb617ac1c042d282873f2c01bf6162 already in progress: creating tablet
I20250115 14:51:50.964776 1164 consensus_queue.cc:260] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:50.964982 888 tablet_service.cc:1467] Processing CreateTablet for tablet c5030b357a804d70b13a93b1d5d5fed1 (DEFAULT_TABLE table=default.Table_07c97d6a492c476291f8fa126fabba8f [id=7fc50307921343ddb6368d736c8adbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.965425 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5030b357a804d70b13a93b1d5d5fed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.966699 1168 raft_consensus.cc:357] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:50.967130 1168 raft_consensus.cc:383] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:50.967231 1168 raft_consensus.cc:738] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:50.967864 1168 consensus_queue.cc:260] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:50.972923 1162 tablet_bootstrap.cc:492] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:50.972996 1025 tablet_service.cc:1467] Processing CreateTablet for tablet c5030b357a804d70b13a93b1d5d5fed1 (DEFAULT_TABLE table=default.Table_07c97d6a492c476291f8fa126fabba8f [id=7fc50307921343ddb6368d736c8adbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.973086 1162 ts_tablet_manager.cc:1397] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.023s
I20250115 14:51:50.973327 1164 ts_tablet_manager.cc:1428] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.053s user 0.009s sys 0.006s
I20250115 14:51:50.973407 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5030b357a804d70b13a93b1d5d5fed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.974097 1162 raft_consensus.cc:357] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:50.974334 1162 raft_consensus.cc:383] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:50.974431 1162 raft_consensus.cc:738] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:50.974671 1162 consensus_queue.cc:260] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:50.975183 1164 tablet_bootstrap.cc:492] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:50.976519 1162 ts_tablet_manager.cc:1428] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250115 14:51:50.976763 1162 tablet_bootstrap.cc:492] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:50.977735 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_02165515be8347129d74a22a54a84ad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:50.978154 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4d560275be034a7cb7e0f17eec823c1e" } new_table_name: "default.Table_7de6e49996a0432d8f8bfa56c51d0782" modify_external_catalogs: true
W20250115 14:51:50.978292 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02165515be8347129d74a22a54a84ad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:50.978390 1162 tablet_bootstrap.cc:654] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:50.979138 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_07c97d6a492c476291f8fa126fabba8f" } new_table_name: "default.Table_5efe2b26efa946d0968c268bda642d94" modify_external_catalogs: true
I20250115 14:51:50.979616 754 tablet_service.cc:1467] Processing CreateTablet for tablet c5030b357a804d70b13a93b1d5d5fed1 (DEFAULT_TABLE table=default.Table_07c97d6a492c476291f8fa126fabba8f [id=7fc50307921343ddb6368d736c8adbfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:50.980221 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_84ae57866f144c9ba1375c09d773bf4f" } new_table_name: "default.Table_8aff0d541b3147b6acb25a0726018b6f" modify_external_catalogs: true
I20250115 14:51:50.980473 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0bf040cafd794a0d92935f720d3e8b09" } new_table_name: "default.Table_575e630b7f8d4823bfa54f3ed4a488fb" modify_external_catalogs: true
I20250115 14:51:50.981739 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_946156b206984caaa8bc03ef6892eff7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:50.983570 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_946156b206984caaa8bc03ef6892eff7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:50.986366 1095 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40227 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:40227 (error 108)
W20250115 14:51:50.987668 848 connection.cc:537] client connection to 127.0.104.60:40227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250115 14:51:50.988667 959 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40227 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250115 14:51:50.992043 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 510ac6d6ee3e488e8b20f3e39a44ab9a consensus state has no leader
W20250115 14:51:50.992566 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5030b357a804d70b13a93b1d5d5fed1 consensus state has no leader
W20250115 14:51:50.992988 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85ab1ccf420d41bd98cfda29caf5a619 has no consensus state
I20250115 14:51:50.993227 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5030b357a804d70b13a93b1d5d5fed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:50.993743 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9460bdb384754c468295752de05d4457" } modify_external_catalogs: true
I20250115 14:51:50.993916 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9460bdb384754c468295752de05d4457" } modify_external_catalogs: true
I20250115 14:51:50.995118 1168 ts_tablet_manager.cc:1428] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.068s user 0.010s sys 0.003s
I20250115 14:51:50.995940 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f04c4734007f45f2b1b6dbe3cfeec857" } modify_external_catalogs: true
I20250115 14:51:50.996137 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f04c4734007f45f2b1b6dbe3cfeec857" } modify_external_catalogs: true
I20250115 14:51:50.996615 1168 tablet_bootstrap.cc:492] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:50.996618 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 17745a9f088946dba78e0630f7400a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:50.997589 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5efe284aa9074f8b841d0ccde93e7c20" } new_table_name: "default.Table_40664b92268d4ba39f3f5852835f97d2" modify_external_catalogs: true
I20250115 14:51:50.998281 1168 tablet_bootstrap.cc:654] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:50.999457 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5d2e74e77f87456396aed66d4d21f840" } modify_external_catalogs: true
I20250115 14:51:50.999621 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5d2e74e77f87456396aed66d4d21f840" } modify_external_catalogs: true
W20250115 14:51:51.000874 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59f25f2f228143c98f6cd6ccb0cf99d7 has no consensus state
I20250115 14:51:51.000207 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fb13e7bf7ead4609a5d43f200ab36cad" } modify_external_catalogs: true
I20250115 14:51:51.001294 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fb13e7bf7ead4609a5d43f200ab36cad" } modify_external_catalogs: true
I20250115 14:51:51.003034 1166 tablet_replica.cc:331] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:51.003567 1166 raft_consensus.cc:2238] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.003994 1166 raft_consensus.cc:2267] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.008551 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 649504c31def4f3f9c30879033b58fbe (table default.Table_536058ab838e47e89d30797e48106518 [id=f6f31e321abb451c877540a31b2fb976]). Expected version 1 got 0
I20250115 14:51:51.010485 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 (DEFAULT_TABLE table=default.Table_7de6e49996a0432d8f8bfa56c51d0782 [id=ff5b7f4d675f496eaf7dd60b0cca7435]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.010731 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 85ab1ccf420d41bd98cfda29caf5a619 (DEFAULT_TABLE table=default.Table_575e630b7f8d4823bfa54f3ed4a488fb [id=af4d0a3d017c4e2cbc04fa416fa68e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.010910 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59f25f2f228143c98f6cd6ccb0cf99d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.011075 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ab1ccf420d41bd98cfda29caf5a619. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.012199 1168 tablet_bootstrap.cc:492] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.012375 1168 ts_tablet_manager.cc:1397] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20250115 14:51:51.010485 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 65f2f961e08846d09df469b74abd5da1 (DEFAULT_TABLE table=default.Table_0181146ed3ba475d86d8e73223c9624b [id=9fa51c9085b244b0ad70a3d49836ea39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.013458 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f2f961e08846d09df469b74abd5da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.013340 1168 raft_consensus.cc:357] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.013530 1168 raft_consensus.cc:383] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.013617 1168 raft_consensus.cc:738] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.013837 1168 consensus_queue.cc:260] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.014338 1168 ts_tablet_manager.cc:1428] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:51.015210 1168 tablet_bootstrap.cc:492] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.017130 753 tablet_service.cc:1467] Processing CreateTablet for tablet 65f2f961e08846d09df469b74abd5da1 (DEFAULT_TABLE table=default.Table_0181146ed3ba475d86d8e73223c9624b [id=9fa51c9085b244b0ad70a3d49836ea39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.017402 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet f1328f038ad3423a8f0edcb13e0f555a (table default.Table_b1b40a84f1874354bf55f4442aa17d8b [id=d7ba4cda40964e1d839efe9e0f1615c0]). Expected version 1 got 0
I20250115 14:51:51.017393 755 tablet_service.cc:1467] Processing CreateTablet for tablet 718f5f0300114292bf838f5f988ab566 (DEFAULT_TABLE table=default.Table_5efe284aa9074f8b841d0ccde93e7c20 [id=da39f1615db74469824c3bf8eb74d260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.017506 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f2f961e08846d09df469b74abd5da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.017746 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 718f5f0300114292bf838f5f988ab566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.018044 752 tablet_service.cc:1467] Processing CreateTablet for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 (DEFAULT_TABLE table=default.Table_7de6e49996a0432d8f8bfa56c51d0782 [id=ff5b7f4d675f496eaf7dd60b0cca7435]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.018373 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59f25f2f228143c98f6cd6ccb0cf99d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.020591 751 tablet_service.cc:1467] Processing CreateTablet for tablet 85ab1ccf420d41bd98cfda29caf5a619 (DEFAULT_TABLE table=default.Table_575e630b7f8d4823bfa54f3ed4a488fb [id=af4d0a3d017c4e2cbc04fa416fa68e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.020881 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ab1ccf420d41bd98cfda29caf5a619. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.023260 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet f1328f038ad3423a8f0edcb13e0f555a (table default.Table_b1b40a84f1874354bf55f4442aa17d8b [id=d7ba4cda40964e1d839efe9e0f1615c0]). Expected version 1 got 0
I20250115 14:51:51.024276 750 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0d566e954045fc85c4fbfe8052b050 (DEFAULT_TABLE table=default.Table_fb13e7bf7ead4609a5d43f200ab36cad [id=a415575b540a4472bc26682438479572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.024622 756 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef5e4d75580465c9c0e36135503f332 (DEFAULT_TABLE table=default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1 [id=546100c082df4fb0984876cf4c60009c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.024832 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d0d566e954045fc85c4fbfe8052b050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.025277 749 tablet_service.cc:1467] Processing CreateTablet for tablet 0e20ad209d7044b09935691f002b9998 (DEFAULT_TABLE table=default.Table_5d2e74e77f87456396aed66d4d21f840 [id=1fc8e792fbbe4faf85699a5a99b99908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.025807 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e20ad209d7044b09935691f002b9998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.026357 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0e20ad209d7044b09935691f002b9998
I20250115 14:51:51.026968 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 101087eee11441f299d5b75e5c80b66b
I20250115 14:51:51.028141 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e20ad209d7044b09935691f002b9998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.028704 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef5e4d75580465c9c0e36135503f332. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.028877 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7249642b69844b6d95fd9c8419e122d6
I20250115 14:51:51.029415 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 101087eee11441f299d5b75e5c80b66b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.031520 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 7249642b69844b6d95fd9c8419e122d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.033166 1166 ts_tablet_manager.cc:1905] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:51.036885 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 718f5f0300114292bf838f5f988ab566 consensus state has no leader
I20250115 14:51:51.039325 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_946156b206984caaa8bc03ef6892eff7" } modify_external_catalogs: true
I20250115 14:51:51.039570 1168 tablet_bootstrap.cc:654] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.039522 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_946156b206984caaa8bc03ef6892eff7" } modify_external_catalogs: true
I20250115 14:51:51.044572 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1d0d566e954045fc85c4fbfe8052b050
I20250115 14:51:51.049518 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 17745a9f088946dba78e0630f7400a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.050045 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 718f5f0300114292bf838f5f988ab566 (DEFAULT_TABLE table=default.Table_5efe284aa9074f8b841d0ccde93e7c20 [id=da39f1615db74469824c3bf8eb74d260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.050477 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 718f5f0300114292bf838f5f988ab566. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.052606 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0d566e954045fc85c4fbfe8052b050 (DEFAULT_TABLE table=default.Table_fb13e7bf7ead4609a5d43f200ab36cad [id=a415575b540a4472bc26682438479572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.052982 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d0d566e954045fc85c4fbfe8052b050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.053499 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 0e20ad209d7044b09935691f002b9998 (DEFAULT_TABLE table=default.Table_5d2e74e77f87456396aed66d4d21f840 [id=1fc8e792fbbe4faf85699a5a99b99908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.053853 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e20ad209d7044b09935691f002b9998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.054867 1164 tablet_bootstrap.cc:654] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.056481 1165 tablet_replica.cc:331] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:51.057050 1165 raft_consensus.cc:2238] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.057495 1165 raft_consensus.cc:2267] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.057982 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e20ad209d7044b09935691f002b9998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.058712 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 101087eee11441f299d5b75e5c80b66b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.058924 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0d566e954045fc85c4fbfe8052b050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.060063 1165 ts_tablet_manager.cc:1905] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.063513 1166 ts_tablet_manager.cc:1918] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.067004 1166 log.cc:1198] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/17745a9f088946dba78e0630f7400a2e
I20250115 14:51:51.067855 1166 ts_tablet_manager.cc:1939] T 17745a9f088946dba78e0630f7400a2e P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:51.068249 1165 ts_tablet_manager.cc:1918] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.068405 1165 log.cc:1198] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/17745a9f088946dba78e0630f7400a2e
I20250115 14:51:51.068820 1165 ts_tablet_manager.cc:1939] T 17745a9f088946dba78e0630f7400a2e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:51:51.072396 1098 negotiation.cc:337] Failed RPC negotiation. Trace:
0115 14:51:50.986957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:40227 (local address 127.0.104.3:36427)
0115 14:51:50.987185 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0115 14:51:50.987205 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0115 14:51:50.987316 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0115 14:51:51.071145 (+ 83829us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":116}
I20250115 14:51:51.073050 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 17745a9f088946dba78e0630f7400a2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.073740 1167 tablet_replica.cc:331] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.073661 888 tablet_service.cc:1467] Processing CreateTablet for tablet 85ab1ccf420d41bd98cfda29caf5a619 (DEFAULT_TABLE table=default.Table_575e630b7f8d4823bfa54f3ed4a488fb [id=af4d0a3d017c4e2cbc04fa416fa68e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.074115 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85ab1ccf420d41bd98cfda29caf5a619. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:51.074112 711 connection.cc:537] client connection to 127.0.104.60:40227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250115 14:51:51.074170 1167 raft_consensus.cc:2238] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:51:51.074676 822 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:40227 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250115 14:51:51.074697 1167 raft_consensus.cc:2267] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.076278 1167 ts_tablet_manager.cc:1905] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.079924 893 tablet_service.cc:1467] Processing CreateTablet for tablet 65f2f961e08846d09df469b74abd5da1 (DEFAULT_TABLE table=default.Table_0181146ed3ba475d86d8e73223c9624b [id=9fa51c9085b244b0ad70a3d49836ea39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.080160 892 tablet_service.cc:1467] Processing CreateTablet for tablet 718f5f0300114292bf838f5f988ab566 (DEFAULT_TABLE table=default.Table_5efe284aa9074f8b841d0ccde93e7c20 [id=da39f1615db74469824c3bf8eb74d260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.080399 887 tablet_service.cc:1467] Processing CreateTablet for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 (DEFAULT_TABLE table=default.Table_7de6e49996a0432d8f8bfa56c51d0782 [id=ff5b7f4d675f496eaf7dd60b0cca7435]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.080657 890 tablet_service.cc:1467] Processing CreateTablet for tablet 0e20ad209d7044b09935691f002b9998 (DEFAULT_TABLE table=default.Table_5d2e74e77f87456396aed66d4d21f840 [id=1fc8e792fbbe4faf85699a5a99b99908]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.080838 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef5e4d75580465c9c0e36135503f332 (DEFAULT_TABLE table=default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1 [id=546100c082df4fb0984876cf4c60009c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.081115 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e20ad209d7044b09935691f002b9998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.081300 889 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0d566e954045fc85c4fbfe8052b050 (DEFAULT_TABLE table=default.Table_fb13e7bf7ead4609a5d43f200ab36cad [id=a415575b540a4472bc26682438479572]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.081504 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 7249642b69844b6d95fd9c8419e122d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.082501 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65f2f961e08846d09df469b74abd5da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.082549 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 101087eee11441f299d5b75e5c80b66b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.082705 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e20ad209d7044b09935691f002b9998. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.082763 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0d566e954045fc85c4fbfe8052b050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.082912 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef5e4d75580465c9c0e36135503f332. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.083093 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 718f5f0300114292bf838f5f988ab566. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:51.083788 1097 tablet.cc:2367] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 14:51:51.083921 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59f25f2f228143c98f6cd6ccb0cf99d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.083916 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 17745a9f088946dba78e0630f7400a2e (table default.Table_4a983082b9b74279b479488150f64d8e [id=6f3d691f400a4cccaf708d491a237b15]) successfully deleted
W20250115 14:51:51.084168 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0e20ad209d7044b09935691f002b9998 because the tablet was not found. No further retry: Not found: Tablet not found: 0e20ad209d7044b09935691f002b9998
I20250115 14:51:51.086220 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d0d566e954045fc85c4fbfe8052b050. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.086141 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef5e4d75580465c9c0e36135503f332 (DEFAULT_TABLE table=default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1 [id=546100c082df4fb0984876cf4c60009c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.086508 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef5e4d75580465c9c0e36135503f332. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:51.085546 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 101087eee11441f299d5b75e5c80b66b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 101087eee11441f299d5b75e5c80b66b already in progress: creating tablet
W20250115 14:51:51.089870 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 1d0d566e954045fc85c4fbfe8052b050 because the tablet was not found. No further retry: Not found: Tablet not found: 1d0d566e954045fc85c4fbfe8052b050
I20250115 14:51:51.091874 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 7249642b69844b6d95fd9c8419e122d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.093178 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0d566e954045fc85c4fbfe8052b050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.098543 1162 tablet_bootstrap.cc:492] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.098721 1162 ts_tablet_manager.cc:1397] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.122s user 0.038s sys 0.063s
I20250115 14:51:51.099463 1162 raft_consensus.cc:357] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.099669 1162 raft_consensus.cc:383] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.099751 1162 raft_consensus.cc:738] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.099993 1162 consensus_queue.cc:260] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.100509 1162 ts_tablet_manager.cc:1428] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:51.102748 1162 tablet_bootstrap.cc:492] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:51.105404 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7249642b69844b6d95fd9c8419e122d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7249642b69844b6d95fd9c8419e122d6 already in progress: creating tablet
I20250115 14:51:51.106297 1162 tablet_bootstrap.cc:654] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:51.114369 961 tablet.cc:2367] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 14:51:51.119619 1168 tablet_bootstrap.cc:492] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.119782 1168 ts_tablet_manager.cc:1397] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.018s
I20250115 14:51:51.120575 1168 raft_consensus.cc:357] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.120777 1168 raft_consensus.cc:383] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.120841 1168 raft_consensus.cc:738] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.121080 1168 consensus_queue.cc:260] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.121565 1168 ts_tablet_manager.cc:1428] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:51.121944 1168 tablet_bootstrap.cc:492] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.123632 1168 tablet_bootstrap.cc:654] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.124275 1167 ts_tablet_manager.cc:1918] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.125735 1167 log.cc:1198] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/17745a9f088946dba78e0630f7400a2e
I20250115 14:51:51.126242 1167 ts_tablet_manager.cc:1939] T 17745a9f088946dba78e0630f7400a2e P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
W20250115 14:51:51.125558 1176 negotiation.cc:337] Failed RPC negotiation. Trace:
0115 14:51:50.999065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:40227 (local address 127.0.104.2:41929)
0115 14:51:51.084691 (+ 85626us) negotiation.cc:107] Waiting for socket to connect
0115 14:51:51.124435 (+ 39744us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":82312,"thread_start_us":73502,"threads_started":1}
I20250115 14:51:51.128945 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 17745a9f088946dba78e0630f7400a2e (table default.Table_4a983082b9b74279b479488150f64d8e [id=6f3d691f400a4cccaf708d491a237b15]) successfully deleted
W20250115 14:51:51.129212 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0e20ad209d7044b09935691f002b9998 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e20ad209d7044b09935691f002b9998 already in progress: creating tablet
W20250115 14:51:51.129421 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7249642b69844b6d95fd9c8419e122d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7249642b69844b6d95fd9c8419e122d6 already in progress: creating tablet
W20250115 14:51:51.129633 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 101087eee11441f299d5b75e5c80b66b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 101087eee11441f299d5b75e5c80b66b already in progress: creating tablet
W20250115 14:51:51.129855 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 1d0d566e954045fc85c4fbfe8052b050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d0d566e954045fc85c4fbfe8052b050 already in progress: creating tablet
W20250115 14:51:51.130093 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:40227, user_credentials={real_user=slave}} blocked reactor thread for 63563.2us
I20250115 14:51:51.130643 1164 tablet_bootstrap.cc:492] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:51.130736 1162 tablet_bootstrap.cc:492] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.130765 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 17745a9f088946dba78e0630f7400a2e (table default.Table_4a983082b9b74279b479488150f64d8e [id=6f3d691f400a4cccaf708d491a237b15]) successfully deleted
I20250115 14:51:51.130897 1162 ts_tablet_manager.cc:1397] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.016s
W20250115 14:51:51.131049 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0e20ad209d7044b09935691f002b9998 because the tablet was not found. No further retry: Not found: Tablet not found: 0e20ad209d7044b09935691f002b9998
I20250115 14:51:51.131104 1164 ts_tablet_manager.cc:1397] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.156s user 0.006s sys 0.002s
W20250115 14:51:51.131315 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 101087eee11441f299d5b75e5c80b66b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 101087eee11441f299d5b75e5c80b66b already in progress: creating tablet
W20250115 14:51:51.131517 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7249642b69844b6d95fd9c8419e122d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7249642b69844b6d95fd9c8419e122d6 already in progress: creating tablet
W20250115 14:51:51.131872 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1d0d566e954045fc85c4fbfe8052b050 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1d0d566e954045fc85c4fbfe8052b050 already in progress: creating tablet
I20250115 14:51:51.131748 1162 raft_consensus.cc:357] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.131999 1162 raft_consensus.cc:383] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.132095 1162 raft_consensus.cc:738] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.132347 1162 consensus_queue.cc:260] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.132354 1164 raft_consensus.cc:357] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.132618 1164 raft_consensus.cc:383] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.132756 1164 raft_consensus.cc:738] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.132879 1162 ts_tablet_manager.cc:1428] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:51.133152 1162 tablet_bootstrap.cc:492] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.133247 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d8489a8ab2964271a090cbea76075ef2
I20250115 14:51:51.136145 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 7bdb617ac1c042d282873f2c01bf6162 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.136734 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 510ac6d6ee3e488e8b20f3e39a44ab9a from {username='slave'} at 127.0.0.1:47464
W20250115 14:51:51.137637 441 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b -> Peer 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227): Couldn't send request to peer 1cc2e23c560f4096be7b14006bdfc525. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:51.137843 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 77eb5ea1789d47eb8a9379923089f687 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.138396 1162 tablet_bootstrap.cc:654] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.138379 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_87a3775a6a0d41eabb46cb6f64e46c0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.138914 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87a3775a6a0d41eabb46cb6f64e46c0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.140666 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 77eb5ea1789d47eb8a9379923089f687 with delete_type TABLET_DATA_DELETED (replaced by tablet 510ac6d6ee3e488e8b20f3e39a44ab9a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.140877 1165 tablet_replica.cc:331] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:51.141055 1165 raft_consensus.cc:2238] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.141181 1165 raft_consensus.cc:2267] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.141263 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 7249642b69844b6d95fd9c8419e122d6 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.141618 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 77eb5ea1789d47eb8a9379923089f687 with delete_type TABLET_DATA_DELETED (replaced by tablet 510ac6d6ee3e488e8b20f3e39a44ab9a) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.141830 1165 ts_tablet_manager.cc:1905] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.142087 1164 consensus_queue.cc:260] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.143790 1164 ts_tablet_manager.cc:1428] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250115 14:51:51.143991 1167 tablet_replica.cc:331] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.144227 1167 raft_consensus.cc:2238] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.147138 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 81a5e53af38c4552ae3cbd953ce57abc (DEFAULT_TABLE table=default.Table_02165515be8347129d74a22a54a84ad9 [id=9db6edcf4cfb491580916c914e61de0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.147426 1164 tablet_bootstrap.cc:492] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:51.147528 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a5e53af38c4552ae3cbd953ce57abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.148046 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ec6d2615d3f348d990f55956ac678e86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.148582 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec6d2615d3f348d990f55956ac678e86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.147054 888 tablet_service.cc:1467] Processing CreateTablet for tablet 81a5e53af38c4552ae3cbd953ce57abc (DEFAULT_TABLE table=default.Table_02165515be8347129d74a22a54a84ad9 [id=9db6edcf4cfb491580916c914e61de0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.149008 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a5e53af38c4552ae3cbd953ce57abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.153384 1165 ts_tablet_manager.cc:1918] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.153522 1165 log.cc:1198] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/77eb5ea1789d47eb8a9379923089f687
I20250115 14:51:51.153877 1165 ts_tablet_manager.cc:1939] T 77eb5ea1789d47eb8a9379923089f687 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:51.154543 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e1192ebeb25e40cbaf8670c6cf0d5f76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.155050 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1192ebeb25e40cbaf8670c6cf0d5f76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.155583 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 77eb5ea1789d47eb8a9379923089f687 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]) successfully deleted
I20250115 14:51:51.155777 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet f1328f038ad3423a8f0edcb13e0f555a (table default.Table_b1b40a84f1874354bf55f4442aa17d8b [id=d7ba4cda40964e1d839efe9e0f1615c0]). Expected version 1 got 0
I20250115 14:51:51.156155 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c788d7ad726246f18eedbc430ca7eb8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.156600 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c788d7ad726246f18eedbc430ca7eb8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.157670 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b876bcff1252445ca4b6881550c9dd82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.158104 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b876bcff1252445ca4b6881550c9dd82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.159188 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0dcb842f12944eb7b2eef007bd137c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:51.159296 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 718f5f0300114292bf838f5f988ab566 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.159190 1146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
W20250115 14:51:51.159632 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dcb842f12944eb7b2eef007bd137c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.159716 1146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: This master's current role is: LEADER
I20250115 14:51:51.161533 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 77eb5ea1789d47eb8a9379923089f687 with delete_type TABLET_DATA_DELETED (replaced by tablet 510ac6d6ee3e488e8b20f3e39a44ab9a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.161715 1168 tablet_bootstrap.cc:492] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.161850 1168 ts_tablet_manager.cc:1397] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.001s
I20250115 14:51:51.164706 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_18add812314a4b0f97f9a61776eb768a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:51.165097 1162 tablet_bootstrap.cc:492] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
W20250115 14:51:51.165155 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18add812314a4b0f97f9a61776eb768a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.165257 1162 ts_tablet_manager.cc:1397] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.014s
I20250115 14:51:51.165799 1168 raft_consensus.cc:357] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:51.165974 1162 raft_consensus.cc:357] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.166184 1162 raft_consensus.cc:383] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.166270 1162 raft_consensus.cc:738] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.166293 1168 raft_consensus.cc:383] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.166440 1168 raft_consensus.cc:738] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
W20250115 14:51:51.166467 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 77eb5ea1789d47eb8a9379923089f687 because the tablet was not found. No further retry: Not found: Tablet not found: 77eb5ea1789d47eb8a9379923089f687
I20250115 14:51:51.166496 1162 consensus_queue.cc:260] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.166761 1168 consensus_queue.cc:260] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:51.167297 1168 ts_tablet_manager.cc:1428] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:51.167191 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b68bb91c80d24bad995dc605e9702d6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.167609 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b68bb91c80d24bad995dc605e9702d6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.167871 1168 tablet_bootstrap.cc:492] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.169549 1168 tablet_bootstrap.cc:654] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.170452 1167 raft_consensus.cc:2267] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.157605 754 tablet_service.cc:1467] Processing CreateTablet for tablet 81a5e53af38c4552ae3cbd953ce57abc (DEFAULT_TABLE table=default.Table_02165515be8347129d74a22a54a84ad9 [id=9db6edcf4cfb491580916c914e61de0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.170526 1162 ts_tablet_manager.cc:1428] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:51.170845 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81a5e53af38c4552ae3cbd953ce57abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.161401 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 7249642b69844b6d95fd9c8419e122d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:51.166604 824 tablet.cc:2367] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250115 14:51:51.172559 1166 tablet_replica.cc:331] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:51.172757 1162 tablet_bootstrap.cc:492] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.173095 1166 raft_consensus.cc:2238] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.173250 1166 raft_consensus.cc:2267] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.173926 1166 ts_tablet_manager.cc:1905] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.173923 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 510ac6d6ee3e488e8b20f3e39a44ab9a
I20250115 14:51:51.174368 1162 tablet_bootstrap.cc:654] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.174378 452 catalog_manager.cc:6596] ReplaceTablet: tablet 510ac6d6ee3e488e8b20f3e39a44ab9a deleted and replaced by tablet bd6d837c64ad4503a224ced67ff1d053
I20250115 14:51:51.175204 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d837c64ad4503a224ced67ff1d053) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:51.175827 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 510ac6d6ee3e488e8b20f3e39a44ab9a already in progress: creating tablet
I20250115 14:51:51.176592 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d837c64ad4503a224ced67ff1d053) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.180327 1162 tablet_bootstrap.cc:492] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.180464 1162 ts_tablet_manager.cc:1397] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250115 14:51:51.181160 1162 raft_consensus.cc:357] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.181345 1162 raft_consensus.cc:383] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.181421 1162 raft_consensus.cc:738] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.181612 1162 consensus_queue.cc:260] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.182117 1162 ts_tablet_manager.cc:1428] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:51.182376 1162 tablet_bootstrap.cc:492] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.184011 1166 ts_tablet_manager.cc:1918] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.184141 1166 log.cc:1198] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/7249642b69844b6d95fd9c8419e122d6
I20250115 14:51:51.184360 1162 tablet_bootstrap.cc:654] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.184512 1166 ts_tablet_manager.cc:1939] T 7249642b69844b6d95fd9c8419e122d6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:51.185425 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 7249642b69844b6d95fd9c8419e122d6 (table default.Table_9460bdb384754c468295752de05d4457 [id=1736b22b006f4ad686507bf204afc585]) successfully deleted
W20250115 14:51:51.185931 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 510ac6d6ee3e488e8b20f3e39a44ab9a already in progress: creating tablet
I20250115 14:51:51.187376 1164 tablet_bootstrap.cc:654] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.188022 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 718f5f0300114292bf838f5f988ab566
I20250115 14:51:51.183707 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d837c64ad4503a224ced67ff1d053) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.188535 449 catalog_manager.cc:6596] ReplaceTablet: tablet 718f5f0300114292bf838f5f988ab566 deleted and replaced by tablet be18f1b6d8a24fabbe818811ef646c62
I20250115 14:51:51.189133 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 718f5f0300114292bf838f5f988ab566 with delete_type TABLET_DATA_DELETED (replaced by tablet be18f1b6d8a24fabbe818811ef646c62) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:51.190574 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 718f5f0300114292bf838f5f988ab566 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 718f5f0300114292bf838f5f988ab566 already in progress: creating tablet
I20250115 14:51:51.190728 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7249642b69844b6d95fd9c8419e122d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.190984 1165 tablet_replica.cc:331] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:51.191152 1165 raft_consensus.cc:2238] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.191315 1165 raft_consensus.cc:2267] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.191846 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 718f5f0300114292bf838f5f988ab566 with delete_type TABLET_DATA_DELETED (replaced by tablet be18f1b6d8a24fabbe818811ef646c62) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.192031 1165 ts_tablet_manager.cc:1905] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.192605 1162 tablet_bootstrap.cc:492] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.192770 1162 ts_tablet_manager.cc:1397] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20250115 14:51:51.192817 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb617ac1c042d282873f2c01bf6162 with delete_type TABLET_DATA_DELETED (replaced by tablet 59f25f2f228143c98f6cd6ccb0cf99d7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.193693 1162 raft_consensus.cc:357] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.193936 1162 raft_consensus.cc:383] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.194021 1162 raft_consensus.cc:738] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.194283 1162 consensus_queue.cc:260] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.194886 1162 ts_tablet_manager.cc:1428] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:51.196370 1162 tablet_bootstrap.cc:492] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.198112 1162 tablet_bootstrap.cc:654] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.198046 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_536058ab838e47e89d30797e48106518" } new_table_name: "default.Table_e3525f8942774391882172dbc88dd1e2" modify_external_catalogs: true
I20250115 14:51:51.198283 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c788d7ad726246f18eedbc430ca7eb8d" } new_table_name: "default.Table_d4858b82cd50473bbd6fda2dcf3bd299" modify_external_catalogs: true
I20250115 14:51:51.202983 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c128b7a02c834993a7068d76854faa7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.203523 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c128b7a02c834993a7068d76854faa7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.204994 1165 ts_tablet_manager.cc:1918] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.205121 1165 log.cc:1198] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/7249642b69844b6d95fd9c8419e122d6
I20250115 14:51:51.205186 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5efe2b26efa946d0968c268bda642d94" } new_table_name: "default.Table_05cc51b3f5ba4d518525c47cbfba4527" modify_external_catalogs: true
I20250115 14:51:51.205510 1165 ts_tablet_manager.cc:1939] T 7249642b69844b6d95fd9c8419e122d6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:51.206432 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 7249642b69844b6d95fd9c8419e122d6 (table default.Table_9460bdb384754c468295752de05d4457 [id=1736b22b006f4ad686507bf204afc585]) successfully deleted
I20250115 14:51:51.207195 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0181146ed3ba475d86d8e73223c9624b" } new_table_name: "default.Table_0fdd97e2341d4405b18d913853a53683" modify_external_catalogs: true
I20250115 14:51:51.207388 1165 tablet_replica.cc:331] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:51.207614 1165 raft_consensus.cc:2238] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.207809 1165 raft_consensus.cc:2267] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:51.208221 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 718f5f0300114292bf838f5f988ab566 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 718f5f0300114292bf838f5f988ab566 already in progress: creating tablet
I20250115 14:51:51.211160 1165 ts_tablet_manager.cc:1905] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.211699 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6f92136d31774ef69d9893b440c2d545"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.212180 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f92136d31774ef69d9893b440c2d545 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:51.214771 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2529350d9264a1793c605ea9d5df9c6 has no consensus state
I20250115 14:51:51.215826 1162 tablet_bootstrap.cc:492] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.216070 1162 ts_tablet_manager.cc:1397] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.005s
I20250115 14:51:51.217015 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 718f5f0300114292bf838f5f988ab566 with delete_type TABLET_DATA_DELETED (replaced by tablet be18f1b6d8a24fabbe818811ef646c62) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.217146 1168 tablet_bootstrap.cc:492] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.216974 1162 raft_consensus.cc:357] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.217304 1168 ts_tablet_manager.cc:1397] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.002s
I20250115 14:51:51.217322 1162 raft_consensus.cc:383] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.217500 1162 raft_consensus.cc:738] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.217803 1162 consensus_queue.cc:260] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.218061 1168 raft_consensus.cc:357] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.218205 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d837c64ad4503a224ced67ff1d053) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.218263 1168 raft_consensus.cc:383] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.218329 1168 raft_consensus.cc:738] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.218544 1168 consensus_queue.cc:260] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.218814 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 649504c31def4f3f9c30879033b58fbe (table default.Table_536058ab838e47e89d30797e48106518 [id=f6f31e321abb451c877540a31b2fb976]). Expected version 1 got 0
W20250115 14:51:51.219228 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 649504c31def4f3f9c30879033b58fbe consensus state has no leader
I20250115 14:51:51.219635 1168 ts_tablet_manager.cc:1428] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:51.219940 1168 tablet_bootstrap.cc:492] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.221500 1162 ts_tablet_manager.cc:1428] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250115 14:51:51.221607 1168 tablet_bootstrap.cc:654] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.223330 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 101087eee11441f299d5b75e5c80b66b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.224543 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 649504c31def4f3f9c30879033b58fbe (table default.Table_e3525f8942774391882172dbc88dd1e2 [id=f6f31e321abb451c877540a31b2fb976]). Expected version 2 got 0
I20250115 14:51:51.224820 1162 tablet_bootstrap.cc:492] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:51.225004 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5030b357a804d70b13a93b1d5d5fed1 consensus state has no leader
W20250115 14:51:51.225601 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65f2f961e08846d09df469b74abd5da1 consensus state has no leader
I20250115 14:51:51.226070 1164 tablet_bootstrap.cc:492] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:51.226243 1164 ts_tablet_manager.cc:1397] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.003s
I20250115 14:51:51.226809 1162 tablet_bootstrap.cc:654] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.226987 1164 raft_consensus.cc:357] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:51.227205 1164 raft_consensus.cc:383] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.227279 1164 raft_consensus.cc:738] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.227519 1164 consensus_queue.cc:260] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:51.228006 1164 ts_tablet_manager.cc:1428] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:51.228272 1164 tablet_bootstrap.cc:492] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:51.228744 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb617ac1c042d282873f2c01bf6162 with delete_type TABLET_DATA_DELETED (replaced by tablet 59f25f2f228143c98f6cd6ccb0cf99d7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.230545 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1" } modify_external_catalogs: true
I20250115 14:51:51.230736 452 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 649504c31def4f3f9c30879033b58fbe (table default.Table_e3525f8942774391882172dbc88dd1e2 [id=f6f31e321abb451c877540a31b2fb976]). Expected version 2 got 0
I20250115 14:51:51.231369 1165 ts_tablet_manager.cc:1918] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.231535 1165 log.cc:1198] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/7bdb617ac1c042d282873f2c01bf6162
I20250115 14:51:51.231494 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7de6e49996a0432d8f8bfa56c51d0782" } modify_external_catalogs: true
I20250115 14:51:51.231657 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7de6e49996a0432d8f8bfa56c51d0782" } modify_external_catalogs: true
I20250115 14:51:51.231860 1165 ts_tablet_manager.cc:1939] T 7bdb617ac1c042d282873f2c01bf6162 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:51.233940 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_05cc51b3f5ba4d518525c47cbfba4527" } modify_external_catalogs: true
I20250115 14:51:51.234133 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_05cc51b3f5ba4d518525c47cbfba4527" } modify_external_catalogs: true
I20250115 14:51:51.235785 1167 ts_tablet_manager.cc:1905] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.230738 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1" } modify_external_catalogs: true
I20250115 14:51:51.237918 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 59f25f2f228143c98f6cd6ccb0cf99d7
I20250115 14:51:51.238179 1164 tablet_bootstrap.cc:654] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.238749 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.239116 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.239135 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.239601 1165 tablet_replica.cc:331] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:51.239778 1165 raft_consensus.cc:2238] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.239931 1165 raft_consensus.cc:2267] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:51.240406 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59f25f2f228143c98f6cd6ccb0cf99d7 already in progress: creating tablet
I20250115 14:51:51.240746 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e3525f8942774391882172dbc88dd1e2" } new_table_name: "default.Table_c1f78195e62c4c148f6b298e2cbc055e" modify_external_catalogs: true
I20250115 14:51:51.242841 1165 ts_tablet_manager.cc:1905] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.243203 1162 tablet_bootstrap.cc:492] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.243431 1162 ts_tablet_manager.cc:1397] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.005s
I20250115 14:51:51.244342 1162 raft_consensus.cc:357] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.244626 1162 raft_consensus.cc:383] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.244776 1164 tablet_bootstrap.cc:492] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:51.244774 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c5030b357a804d70b13a93b1d5d5fed1
I20250115 14:51:51.244758 1162 raft_consensus.cc:738] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.244911 1164 ts_tablet_manager.cc:1397] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20250115 14:51:51.245186 1162 consensus_queue.cc:260] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.245875 888 tablet_service.cc:1514] Processing DeleteTablet for tablet c5030b357a804d70b13a93b1d5d5fed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.245985 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet c5030b357a804d70b13a93b1d5d5fed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.249295 1162 ts_tablet_manager.cc:1428] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20250115 14:51:51.251070 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 649504c31def4f3f9c30879033b58fbe consensus state has no leader
I20250115 14:51:51.251276 754 tablet_service.cc:1514] Processing DeleteTablet for tablet c5030b357a804d70b13a93b1d5d5fed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.251778 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3ef5e4d75580465c9c0e36135503f332
I20250115 14:51:51.252385 1162 tablet_bootstrap.cc:492] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:51.252468 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c5030b357a804d70b13a93b1d5d5fed1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5030b357a804d70b13a93b1d5d5fed1 already in progress: creating tablet
I20250115 14:51:51.252919 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef5e4d75580465c9c0e36135503f332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:51.253405 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3ef5e4d75580465c9c0e36135503f332 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef5e4d75580465c9c0e36135503f332 already in progress: creating tablet
I20250115 14:51:51.253713 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef5e4d75580465c9c0e36135503f332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.254761 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef5e4d75580465c9c0e36135503f332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.254967 1162 tablet_bootstrap.cc:654] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.255745 1164 raft_consensus.cc:357] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.255959 1164 raft_consensus.cc:383] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.256050 1164 raft_consensus.cc:738] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.256273 1164 consensus_queue.cc:260] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.256747 1168 tablet_bootstrap.cc:492] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.256937 1168 ts_tablet_manager.cc:1397] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.004s
I20250115 14:51:51.257777 1168 raft_consensus.cc:357] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.258024 1168 raft_consensus.cc:383] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.258155 1168 raft_consensus.cc:738] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.258411 1168 consensus_queue.cc:260] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.259089 1168 ts_tablet_manager.cc:1428] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:51.259479 1168 tablet_bootstrap.cc:492] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.260629 1164 ts_tablet_manager.cc:1428] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20250115 14:51:51.261317 1168 tablet_bootstrap.cc:654] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.262876 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0fdd97e2341d4405b18d913853a53683" } modify_external_catalogs: true
I20250115 14:51:51.263069 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0fdd97e2341d4405b18d913853a53683" } modify_external_catalogs: true
I20250115 14:51:51.263916 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 77eb5ea1789d47eb8a9379923089f687 with delete_type TABLET_DATA_DELETED (replaced by tablet 510ac6d6ee3e488e8b20f3e39a44ab9a) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.264189 1166 tablet_replica.cc:331] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:51.264372 1166 raft_consensus.cc:2238] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.264547 1166 raft_consensus.cc:2267] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.266265 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 7249642b69844b6d95fd9c8419e122d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.267083 1162 tablet_bootstrap.cc:492] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.267237 1162 ts_tablet_manager.cc:1397] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250115 14:51:51.267643 1166 ts_tablet_manager.cc:1905] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.268007 1162 raft_consensus.cc:357] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.268206 1162 raft_consensus.cc:383] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.268321 1162 raft_consensus.cc:738] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.268551 1162 consensus_queue.cc:260] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.269845 1162 ts_tablet_manager.cc:1428] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:51.270787 1162 tablet_bootstrap.cc:492] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.271080 754 tablet_service.cc:1467] Processing CreateTablet for tablet 379087c496d5401ebcbd1469ce1f68f3 (DEFAULT_TABLE table=default.Table_b68bb91c80d24bad995dc605e9702d6f [id=959c2861880b40dcbb3780f3dff3b0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.271437 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 379087c496d5401ebcbd1469ce1f68f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.272739 1162 tablet_bootstrap.cc:654] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.273370 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 379087c496d5401ebcbd1469ce1f68f3 (DEFAULT_TABLE table=default.Table_b68bb91c80d24bad995dc605e9702d6f [id=959c2861880b40dcbb3780f3dff3b0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.278168 888 tablet_service.cc:1467] Processing CreateTablet for tablet 379087c496d5401ebcbd1469ce1f68f3 (DEFAULT_TABLE table=default.Table_b68bb91c80d24bad995dc605e9702d6f [id=959c2861880b40dcbb3780f3dff3b0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.279024 1165 ts_tablet_manager.cc:1918] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.279165 1165 log.cc:1198] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/510ac6d6ee3e488e8b20f3e39a44ab9a
I20250115 14:51:51.279503 1165 ts_tablet_manager.cc:1939] T 510ac6d6ee3e488e8b20f3e39a44ab9a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:51.279884 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 379087c496d5401ebcbd1469ce1f68f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.280619 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 379087c496d5401ebcbd1469ce1f68f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.280768 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f2f961e08846d09df469b74abd5da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.281109 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 65f2f961e08846d09df469b74abd5da1
I20250115 14:51:51.282168 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f2f961e08846d09df469b74abd5da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.282557 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f2f961e08846d09df469b74abd5da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.283419 749 tablet_service.cc:1467] Processing CreateTablet for tablet edd50ffca1ee49d0bbe96187480ea26b (DEFAULT_TABLE table=default.Table_ec6d2615d3f348d990f55956ac678e86 [id=f82deda2142247809c0f6a3fc2edf5c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.284682 1027 tablet_service.cc:1467] Processing CreateTablet for tablet edd50ffca1ee49d0bbe96187480ea26b (DEFAULT_TABLE table=default.Table_ec6d2615d3f348d990f55956ac678e86 [id=f82deda2142247809c0f6a3fc2edf5c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.285043 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd50ffca1ee49d0bbe96187480ea26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.286155 1165 tablet_replica.cc:331] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:51.286355 1165 raft_consensus.cc:2238] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.286518 1165 raft_consensus.cc:2267] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.287158 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 510ac6d6ee3e488e8b20f3e39a44ab9a (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]) successfully deleted
I20250115 14:51:51.287290 893 tablet_service.cc:1467] Processing CreateTablet for tablet edd50ffca1ee49d0bbe96187480ea26b (DEFAULT_TABLE table=default.Table_ec6d2615d3f348d990f55956ac678e86 [id=f82deda2142247809c0f6a3fc2edf5c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.287580 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f2f961e08846d09df469b74abd5da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.287645 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd50ffca1ee49d0bbe96187480ea26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.287897 1167 ts_tablet_manager.cc:1918] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.288018 1167 log.cc:1198] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/77eb5ea1789d47eb8a9379923089f687
I20250115 14:51:51.288431 1167 ts_tablet_manager.cc:1939] T 77eb5ea1789d47eb8a9379923089f687 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.290374 1165 ts_tablet_manager.cc:1905] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.293329 756 tablet_service.cc:1467] Processing CreateTablet for tablet e115bdfa1d69421890122f9afdc23b29 (DEFAULT_TABLE table=default.Table_e1192ebeb25e40cbaf8670c6cf0d5f76 [id=1c779c8a53f54707abe1d619b30a912e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.293783 888 tablet_service.cc:1467] Processing CreateTablet for tablet e115bdfa1d69421890122f9afdc23b29 (DEFAULT_TABLE table=default.Table_e1192ebeb25e40cbaf8670c6cf0d5f76 [id=1c779c8a53f54707abe1d619b30a912e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.294083 755 tablet_service.cc:1467] Processing CreateTablet for tablet f8346c87744f46daa623c414e7307299 (DEFAULT_TABLE table=default.Table_18add812314a4b0f97f9a61776eb768a [id=c85266bab3cd4586a91a82c1f7f96c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.294162 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e115bdfa1d69421890122f9afdc23b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.294927 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbd41d946d245b78e6686f8fae55449 (DEFAULT_TABLE table=default.Table_c128b7a02c834993a7068d76854faa7d [id=4745e3d5a5ba4176ad2b4f72b09846c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.295365 750 tablet_service.cc:1467] Processing CreateTablet for tablet d2529350d9264a1793c605ea9d5df9c6 (DEFAULT_TABLE table=default.Table_d4858b82cd50473bbd6fda2dcf3bd299 [id=0d2a48bbde7045e69429c5464157ab08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.296051 752 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4259dfb8ee4b618c0bacc910addd0a (DEFAULT_TABLE table=default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.296521 753 tablet_service.cc:1467] Processing CreateTablet for tablet 70e2495496b746abb568e78377c83d1e (DEFAULT_TABLE table=default.Table_0dcb842f12944eb7b2eef007bd137c72 [id=017270e9a3ad41f18c34b3cfc2c4cd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.296672 1167 tablet_replica.cc:331] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.296890 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 77eb5ea1789d47eb8a9379923089f687 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]) successfully deleted
I20250115 14:51:51.297420 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f8346c87744f46daa623c414e7307299 (DEFAULT_TABLE table=default.Table_18add812314a4b0f97f9a61776eb768a [id=c85266bab3cd4586a91a82c1f7f96c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.297884 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbd41d946d245b78e6686f8fae55449 (DEFAULT_TABLE table=default.Table_c128b7a02c834993a7068d76854faa7d [id=4745e3d5a5ba4176ad2b4f72b09846c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.297865 887 tablet_service.cc:1467] Processing CreateTablet for tablet f8346c87744f46daa623c414e7307299 (DEFAULT_TABLE table=default.Table_18add812314a4b0f97f9a61776eb768a [id=c85266bab3cd4586a91a82c1f7f96c09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.298198 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8346c87744f46daa623c414e7307299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.298314 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d2529350d9264a1793c605ea9d5df9c6 (DEFAULT_TABLE table=default.Table_d4858b82cd50473bbd6fda2dcf3bd299 [id=0d2a48bbde7045e69429c5464157ab08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.298571 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b4259dfb8ee4b618c0bacc910addd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.298732 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4259dfb8ee4b618c0bacc910addd0a (DEFAULT_TABLE table=default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.298923 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2529350d9264a1793c605ea9d5df9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.299110 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e2495496b746abb568e78377c83d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.299273 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbd41d946d245b78e6686f8fae55449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.299669 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8346c87744f46daa623c414e7307299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.300063 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e115bdfa1d69421890122f9afdc23b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.300514 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd50ffca1ee49d0bbe96187480ea26b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.302112 892 tablet_service.cc:1467] Processing CreateTablet for tablet 2b4259dfb8ee4b618c0bacc910addd0a (DEFAULT_TABLE table=default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.302456 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b4259dfb8ee4b618c0bacc910addd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.304010 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9bbd41d946d245b78e6686f8fae55449 (DEFAULT_TABLE table=default.Table_c128b7a02c834993a7068d76854faa7d [id=4745e3d5a5ba4176ad2b4f72b09846c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.304347 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbd41d946d245b78e6686f8fae55449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.308324 889 tablet_service.cc:1467] Processing CreateTablet for tablet d2529350d9264a1793c605ea9d5df9c6 (DEFAULT_TABLE table=default.Table_d4858b82cd50473bbd6fda2dcf3bd299 [id=0d2a48bbde7045e69429c5464157ab08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.308681 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2529350d9264a1793c605ea9d5df9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.296855 1167 raft_consensus.cc:2238] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.310882 1167 raft_consensus.cc:2267] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:51.310860 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 510ac6d6ee3e488e8b20f3e39a44ab9a already in progress: creating tablet
W20250115 14:51:51.311103 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 718f5f0300114292bf838f5f988ab566 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 718f5f0300114292bf838f5f988ab566 already in progress: creating tablet
I20250115 14:51:51.311897 1027 tablet_service.cc:1467] Processing CreateTablet for tablet e115bdfa1d69421890122f9afdc23b29 (DEFAULT_TABLE table=default.Table_e1192ebeb25e40cbaf8670c6cf0d5f76 [id=1c779c8a53f54707abe1d619b30a912e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.312650 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2529350d9264a1793c605ea9d5df9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.313146 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bbd41d946d245b78e6686f8fae55449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.313572 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b4259dfb8ee4b618c0bacc910addd0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.313974 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8346c87744f46daa623c414e7307299. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.314944 1166 ts_tablet_manager.cc:1918] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.315074 1166 log.cc:1198] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/77eb5ea1789d47eb8a9379923089f687
I20250115 14:51:51.315490 1166 ts_tablet_manager.cc:1939] T 77eb5ea1789d47eb8a9379923089f687 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:51.316135 1164 tablet_bootstrap.cc:492] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:51.317793 1164 tablet_bootstrap.cc:654] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.318311 1167 ts_tablet_manager.cc:1905] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.320251 1168 tablet_bootstrap.cc:492] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.320470 1168 ts_tablet_manager.cc:1397] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.004s
I20250115 14:51:51.320928 1164 tablet_bootstrap.cc:492] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:51.321090 1164 ts_tablet_manager.cc:1397] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250115 14:51:51.321228 1168 raft_consensus.cc:357] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.321440 1168 raft_consensus.cc:383] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.321511 1168 raft_consensus.cc:738] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.321739 1168 consensus_queue.cc:260] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.322232 1168 ts_tablet_manager.cc:1428] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:51.321868 1164 raft_consensus.cc:357] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.322500 1164 raft_consensus.cc:383] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.322590 1164 raft_consensus.cc:738] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.322868 1164 consensus_queue.cc:260] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.324530 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 70e2495496b746abb568e78377c83d1e (DEFAULT_TABLE table=default.Table_0dcb842f12944eb7b2eef007bd137c72 [id=017270e9a3ad41f18c34b3cfc2c4cd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.324787 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 715274a681f447c594e8fcd38d6fd609 (DEFAULT_TABLE table=default.Table_87a3775a6a0d41eabb46cb6f64e46c0d [id=d1df7ebdc5994a36800250155aefd024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.325182 889 tablet_service.cc:1467] Processing CreateTablet for tablet 70e2495496b746abb568e78377c83d1e (DEFAULT_TABLE table=default.Table_0dcb842f12944eb7b2eef007bd137c72 [id=017270e9a3ad41f18c34b3cfc2c4cd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.325518 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 66c6c392518a464dab664e5f9321f1f6 (DEFAULT_TABLE table=default.Table_6f92136d31774ef69d9893b440c2d545 [id=52b6d8da69f74139a394cafe66e48a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.325573 890 tablet_service.cc:1467] Processing CreateTablet for tablet 715274a681f447c594e8fcd38d6fd609 (DEFAULT_TABLE table=default.Table_87a3775a6a0d41eabb46cb6f64e46c0d [id=d1df7ebdc5994a36800250155aefd024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.326128 1164 ts_tablet_manager.cc:1428] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:51.326085 888 tablet_service.cc:1467] Processing CreateTablet for tablet 66c6c392518a464dab664e5f9321f1f6 (DEFAULT_TABLE table=default.Table_6f92136d31774ef69d9893b440c2d545 [id=52b6d8da69f74139a394cafe66e48a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.327224 1164 tablet_bootstrap.cc:492] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:51.328943 1164 tablet_bootstrap.cc:654] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.336200 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 715274a681f447c594e8fcd38d6fd609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.336630 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdb617ac1c042d282873f2c01bf6162 with delete_type TABLET_DATA_DELETED (replaced by tablet 59f25f2f228143c98f6cd6ccb0cf99d7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.336617 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_40664b92268d4ba39f3f5852835f97d2" } modify_external_catalogs: true
I20250115 14:51:51.336824 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_40664b92268d4ba39f3f5852835f97d2" } modify_external_catalogs: true
I20250115 14:51:51.337688 753 tablet_service.cc:1467] Processing CreateTablet for tablet 66c6c392518a464dab664e5f9321f1f6 (DEFAULT_TABLE table=default.Table_6f92136d31774ef69d9893b440c2d545 [id=52b6d8da69f74139a394cafe66e48a35]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.338052 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c6c392518a464dab664e5f9321f1f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.339462 1166 tablet_replica.cc:331] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:51.339650 1166 raft_consensus.cc:2238] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.339825 1166 raft_consensus.cc:2267] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.340274 1168 tablet_bootstrap.cc:492] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.344051 1168 tablet_bootstrap.cc:654] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.344448 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d837c64ad4503a224ced67ff1d053) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.346062 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c6c392518a464dab664e5f9321f1f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.346302 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e2495496b746abb568e78377c83d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.348603 447 master_service.cc:946] ReplaceTablet: received request to replace tablet f1328f038ad3423a8f0edcb13e0f555a from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.348747 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 77eb5ea1789d47eb8a9379923089f687 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]) successfully deleted
W20250115 14:51:51.348984 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 65f2f961e08846d09df469b74abd5da1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65f2f961e08846d09df469b74abd5da1 already in progress: creating tablet
I20250115 14:51:51.349439 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 59f25f2f228143c98f6cd6ccb0cf99d7 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.351060 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet be18f1b6d8a24fabbe818811ef646c62
I20250115 14:51:51.353302 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet be18f1b6d8a24fabbe818811ef646c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.353559 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e2495496b746abb568e78377c83d1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.353955 892 tablet_service.cc:1514] Processing DeleteTablet for tablet be18f1b6d8a24fabbe818811ef646c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.354132 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 715274a681f447c594e8fcd38d6fd609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.354545 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66c6c392518a464dab664e5f9321f1f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.359921 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f1328f038ad3423a8f0edcb13e0f555a
I20250115 14:51:51.360463 447 catalog_manager.cc:6596] ReplaceTablet: tablet f1328f038ad3423a8f0edcb13e0f555a deleted and replaced by tablet 22edaa6e70d942ba98777973476f449d
I20250115 14:51:51.361104 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f1328f038ad3423a8f0edcb13e0f555a with delete_type TABLET_DATA_DELETED (replaced by tablet 22edaa6e70d942ba98777973476f449d) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.361649 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f1328f038ad3423a8f0edcb13e0f555a with delete_type TABLET_DATA_DELETED (replaced by tablet 22edaa6e70d942ba98777973476f449d) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.366748 1167 ts_tablet_manager.cc:1918] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.366868 1167 log.cc:1198] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/7bdb617ac1c042d282873f2c01bf6162
I20250115 14:51:51.367246 1167 ts_tablet_manager.cc:1939] T 7bdb617ac1c042d282873f2c01bf6162 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
W20250115 14:51:51.367967 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59f25f2f228143c98f6cd6ccb0cf99d7 already in progress: creating tablet
I20250115 14:51:51.368124 1167 tablet_replica.cc:331] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.368323 1167 raft_consensus.cc:2238] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.368479 1167 raft_consensus.cc:2267] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:51.368525 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c5030b357a804d70b13a93b1d5d5fed1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c5030b357a804d70b13a93b1d5d5fed1 already in progress: creating tablet
W20250115 14:51:51.368746 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3ef5e4d75580465c9c0e36135503f332 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef5e4d75580465c9c0e36135503f332 already in progress: creating tablet
I20250115 14:51:51.369560 1167 ts_tablet_manager.cc:1905] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.370316 1168 tablet_bootstrap.cc:492] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.370455 1168 ts_tablet_manager.cc:1397] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.001s
I20250115 14:51:51.370466 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 101087eee11441f299d5b75e5c80b66b from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.364987 1165 ts_tablet_manager.cc:1918] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.371203 1168 raft_consensus.cc:357] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.371397 1168 raft_consensus.cc:383] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.371338 1165 log.cc:1198] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/101087eee11441f299d5b75e5c80b66b
I20250115 14:51:51.371484 1168 raft_consensus.cc:738] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.371724 1168 consensus_queue.cc:260] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.373106 1165 ts_tablet_manager.cc:1939] T 101087eee11441f299d5b75e5c80b66b P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:51.375541 1168 ts_tablet_manager.cc:1428] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:51.375805 1168 tablet_bootstrap.cc:492] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.377633 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 101087eee11441f299d5b75e5c80b66b (table default.Table_f04c4734007f45f2b1b6dbe3cfeec857 [id=05c9f91a89fb43c8918c6cfe74360e43]) successfully deleted
I20250115 14:51:51.380714 1168 tablet_bootstrap.cc:654] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.381161 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cf6ef8fd0af84c5fa5e257ace88f2a7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.381711 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf6ef8fd0af84c5fa5e257ace88f2a7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.352949 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e115bdfa1d69421890122f9afdc23b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.382719 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 65f2f961e08846d09df469b74abd5da1 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.384183 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 101087eee11441f299d5b75e5c80b66b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.387053 1167 ts_tablet_manager.cc:1918] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.387235 1167 log.cc:1198] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/7249642b69844b6d95fd9c8419e122d6
I20250115 14:51:51.387598 1167 ts_tablet_manager.cc:1939] T 7249642b69844b6d95fd9c8419e122d6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.388516 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 7249642b69844b6d95fd9c8419e122d6 (table default.Table_9460bdb384754c468295752de05d4457 [id=1736b22b006f4ad686507bf204afc585]) successfully deleted
W20250115 14:51:51.389199 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 65f2f961e08846d09df469b74abd5da1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65f2f961e08846d09df469b74abd5da1 already in progress: creating tablet
I20250115 14:51:51.389566 1167 tablet_replica.cc:331] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.389773 1167 raft_consensus.cc:2238] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.390304 1174 raft_consensus.cc:491] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:51.390523 1174 raft_consensus.cc:513] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.392449 1174 leader_election.cc:290] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:51.394649 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8003836a7a5c4f79a7bec455f2d8a325"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:51.394709 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a78c10f9e6384d7d89d3add6f1371429"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.395269 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8003836a7a5c4f79a7bec455f2d8a325 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:51.395367 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a78c10f9e6384d7d89d3add6f1371429 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.396608 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_63d814458ff041bc8d15d1cdef48de89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.397089 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63d814458ff041bc8d15d1cdef48de89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.397058 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_64abfdac91a54d6bb59ccfc7791a170c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.397497 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64abfdac91a54d6bb59ccfc7791a170c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.398195 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_733144094e85412998e706e7a0f93dbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.398653 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_733144094e85412998e706e7a0f93dbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.400151 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_319faebd004b4a1e93821d947ef288d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.400574 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_319faebd004b4a1e93821d947ef288d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.401773 1167 raft_consensus.cc:2267] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.402725 1167 ts_tablet_manager.cc:1905] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.403554 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b4259dfb8ee4b618c0bacc910addd0a from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.394693 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_662e371503c24e6cacf91cb2afe19d53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.405174 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_662e371503c24e6cacf91cb2afe19d53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:51.410269 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59f25f2f228143c98f6cd6ccb0cf99d7 already in progress: creating tablet
I20250115 14:51:51.412824 889 tablet_service.cc:1467] Processing CreateTablet for tablet bd6d837c64ad4503a224ced67ff1d053 (DEFAULT_TABLE table=default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.413223 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6d837c64ad4503a224ced67ff1d053. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.414650 1026 tablet_service.cc:1467] Processing CreateTablet for tablet bd6d837c64ad4503a224ced67ff1d053 (DEFAULT_TABLE table=default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.414885 1166 ts_tablet_manager.cc:1905] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.415019 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6d837c64ad4503a224ced67ff1d053. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.416564 888 tablet_service.cc:1467] Processing CreateTablet for tablet be18f1b6d8a24fabbe818811ef646c62 (DEFAULT_TABLE table=default.Table_40664b92268d4ba39f3f5852835f97d2 [id=da39f1615db74469824c3bf8eb74d260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.416880 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be18f1b6d8a24fabbe818811ef646c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.417438 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cf6ef8fd0af84c5fa5e257ace88f2a7b" } new_table_name: "default.Table_dd152d863f394374a58bf845d3d259e2" modify_external_catalogs: true
I20250115 14:51:51.417887 752 tablet_service.cc:1514] Processing DeleteTablet for tablet be18f1b6d8a24fabbe818811ef646c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.418643 754 tablet_service.cc:1467] Processing CreateTablet for tablet 715274a681f447c594e8fcd38d6fd609 (DEFAULT_TABLE table=default.Table_87a3775a6a0d41eabb46cb6f64e46c0d [id=d1df7ebdc5994a36800250155aefd024]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.420629 1028 tablet_service.cc:1467] Processing CreateTablet for tablet be18f1b6d8a24fabbe818811ef646c62 (DEFAULT_TABLE table=default.Table_40664b92268d4ba39f3f5852835f97d2 [id=da39f1615db74469824c3bf8eb74d260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.421000 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be18f1b6d8a24fabbe818811ef646c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.423445 1162 tablet_bootstrap.cc:492] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.423607 1162 ts_tablet_manager.cc:1397] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.153s user 0.034s sys 0.068s
I20250115 14:51:51.424419 1162 raft_consensus.cc:357] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.424666 1162 raft_consensus.cc:383] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.424759 1162 raft_consensus.cc:738] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.425011 1162 consensus_queue.cc:260] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.425647 1162 ts_tablet_manager.cc:1428] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:51.425933 1162 tablet_bootstrap.cc:492] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.427539 1162 tablet_bootstrap.cc:654] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.431782 1164 tablet_bootstrap.cc:492] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:51.431926 1164 ts_tablet_manager.cc:1397] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.105s user 0.013s sys 0.002s
I20250115 14:51:51.433945 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3529669cc14746fda74ad45528cbcf4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:51.434351 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.434276 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0dcb842f12944eb7b2eef007bd137c72" } new_table_name: "default.Table_3921752ce1634998826bf8a3513582d9" modify_external_catalogs: true
W20250115 14:51:51.434542 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3529669cc14746fda74ad45528cbcf4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.435510 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b1b40a84f1874354bf55f4442aa17d8b" } new_table_name: "default.Table_8d7e6f90e6a64f5d84f728b16b2deea8" modify_external_catalogs: true
I20250115 14:51:51.435631 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1328f038ad3423a8f0edcb13e0f555a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:51.435840 1166 ts_tablet_manager.cc:1918] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.435979 1166 log.cc:1198] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/7bdb617ac1c042d282873f2c01bf6162
I20250115 14:51:51.436156 1049 raft_consensus.cc:2463] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:51.436312 1166 ts_tablet_manager.cc:1939] T 7bdb617ac1c042d282873f2c01bf6162 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:51.436441 1162 tablet_bootstrap.cc:492] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.436565 1162 ts_tablet_manager.cc:1397] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250115 14:51:51.436594 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 715274a681f447c594e8fcd38d6fd609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.436606 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_575e630b7f8d4823bfa54f3ed4a488fb" } new_table_name: "default.Table_eb8583ed6e0c445b9883e8391d53b878" modify_external_catalogs: true
I20250115 14:51:51.437301 847 leader_election.cc:304] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:51.437340 1162 raft_consensus.cc:357] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.437502 1162 raft_consensus.cc:383] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.437573 1162 raft_consensus.cc:738] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.437709 1174 raft_consensus.cc:2798] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:51.437646 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_65ba1d73ba754551a967823306bb4d16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:51.437825 1174 raft_consensus.cc:491] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:51.437750 1162 consensus_queue.cc:260] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.437904 1174 raft_consensus.cc:3054] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:51.438133 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65ba1d73ba754551a967823306bb4d16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.438355 1162 ts_tablet_manager.cc:1428] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:51.438757 1164 raft_consensus.cc:357] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.438951 1164 raft_consensus.cc:383] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.439026 1164 raft_consensus.cc:738] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.439257 1164 consensus_queue.cc:260] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.440479 1164 ts_tablet_manager.cc:1428] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250115 14:51:51.440738 1164 tablet_bootstrap.cc:492] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:51.442452 1164 tablet_bootstrap.cc:654] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.443682 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2b4259dfb8ee4b618c0bacc910addd0a
I20250115 14:51:51.444157 447 catalog_manager.cc:6596] ReplaceTablet: tablet 2b4259dfb8ee4b618c0bacc910addd0a deleted and replaced by tablet 3a38a6bc32314396b497e7ae166a63f7
I20250115 14:51:51.444665 1174 raft_consensus.cc:513] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.445222 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_87a3775a6a0d41eabb46cb6f64e46c0d" } modify_external_catalogs: true
I20250115 14:51:51.445348 1174 leader_election.cc:290] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:51.445418 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_87a3775a6a0d41eabb46cb6f64e46c0d" } modify_external_catalogs: true
I20250115 14:51:51.445921 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4259dfb8ee4b618c0bacc910addd0a with delete_type TABLET_DATA_DELETED (replaced by tablet 3a38a6bc32314396b497e7ae166a63f7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.446486 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4259dfb8ee4b618c0bacc910addd0a with delete_type TABLET_DATA_DELETED (replaced by tablet 3a38a6bc32314396b497e7ae166a63f7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.447116 1162 tablet_bootstrap.cc:492] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.447978 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4259dfb8ee4b618c0bacc910addd0a with delete_type TABLET_DATA_DELETED (replaced by tablet 3a38a6bc32314396b497e7ae166a63f7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.448583 753 tablet_service.cc:1467] Processing CreateTablet for tablet be18f1b6d8a24fabbe818811ef646c62 (DEFAULT_TABLE table=default.Table_40664b92268d4ba39f3f5852835f97d2 [id=da39f1615db74469824c3bf8eb74d260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.448724 1162 tablet_bootstrap.cc:654] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.449083 750 tablet_service.cc:1467] Processing CreateTablet for tablet bd6d837c64ad4503a224ced67ff1d053 (DEFAULT_TABLE table=default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.450163 756 tablet_service.cc:1514] Processing DeleteTablet for tablet f1328f038ad3423a8f0edcb13e0f555a with delete_type TABLET_DATA_DELETED (replaced by tablet 22edaa6e70d942ba98777973476f449d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.452503 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1328f038ad3423a8f0edcb13e0f555a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:51.452692 1049 raft_consensus.cc:3054] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:51.453856 1167 ts_tablet_manager.cc:1918] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.454108 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c128b7a02c834993a7068d76854faa7d" } modify_external_catalogs: true
I20250115 14:51:51.454195 1167 log.cc:1198] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/510ac6d6ee3e488e8b20f3e39a44ab9a
I20250115 14:51:51.454301 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c128b7a02c834993a7068d76854faa7d" } modify_external_catalogs: true
I20250115 14:51:51.454527 1167 ts_tablet_manager.cc:1939] T 510ac6d6ee3e488e8b20f3e39a44ab9a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.455390 1049 raft_consensus.cc:2463] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:51.455873 847 leader_election.cc:304] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:51.456135 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 510ac6d6ee3e488e8b20f3e39a44ab9a (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]) successfully deleted
I20250115 14:51:51.456173 1174 raft_consensus.cc:2798] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:51:51.456979 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22edaa6e70d942ba98777973476f449d has no consensus state
I20250115 14:51:51.457288 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_63d814458ff041bc8d15d1cdef48de89" } new_table_name: "default.Table_6dfc58433fe64bf48f2215d6caab2368" modify_external_catalogs: true
I20250115 14:51:51.458057 1168 tablet_bootstrap.cc:492] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.458221 1168 ts_tablet_manager.cc:1397] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.082s user 0.001s sys 0.006s
W20250115 14:51:51.458532 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c322301320ba425a94d7321c08f7f9e6 has no consensus state
W20250115 14:51:51.458927 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70e2495496b746abb568e78377c83d1e consensus state has no leader
I20250115 14:51:51.458949 1168 raft_consensus.cc:357] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.459167 1168 raft_consensus.cc:383] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.459239 1168 raft_consensus.cc:738] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.459468 1168 consensus_queue.cc:260] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:51.460670 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet be18f1b6d8a24fabbe818811ef646c62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be18f1b6d8a24fabbe818811ef646c62 already in progress: creating tablet
I20250115 14:51:51.461225 1165 tablet_replica.cc:331] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:51.461417 1165 raft_consensus.cc:2238] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.461601 1165 raft_consensus.cc:2267] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.462558 1165 ts_tablet_manager.cc:1905] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.467970 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_18add812314a4b0f97f9a61776eb768a" } modify_external_catalogs: true
I20250115 14:51:51.468168 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_18add812314a4b0f97f9a61776eb768a" } modify_external_catalogs: true
I20250115 14:51:51.468472 1167 tablet_replica.cc:331] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.468822 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be18f1b6d8a24fabbe818811ef646c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.469703 1162 tablet_bootstrap.cc:492] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.469859 1162 ts_tablet_manager.cc:1397] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20250115 14:51:51.470606 1162 raft_consensus.cc:357] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.470798 1162 raft_consensus.cc:383] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.470866 1162 raft_consensus.cc:738] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.471062 1174 raft_consensus.cc:695] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:51.471477 1174 consensus_queue.cc:237] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.472595 1168 ts_tablet_manager.cc:1428] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250115 14:51:51.473021 1168 tablet_bootstrap.cc:492] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.474514 1162 consensus_queue.cc:260] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.475006 1162 ts_tablet_manager.cc:1428] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:51:51.475327 1162 tablet_bootstrap.cc:492] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.477109 1162 tablet_bootstrap.cc:654] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.478497 1168 tablet_bootstrap.cc:654] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.481010 1162 tablet_bootstrap.cc:492] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.481155 1162 ts_tablet_manager.cc:1397] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250115 14:51:51.481491 888 tablet_service.cc:1467] Processing CreateTablet for tablet 22edaa6e70d942ba98777973476f449d (DEFAULT_TABLE table=default.Table_8d7e6f90e6a64f5d84f728b16b2deea8 [id=d7ba4cda40964e1d839efe9e0f1615c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.481865 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22edaa6e70d942ba98777973476f449d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.481855 1162 raft_consensus.cc:357] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.482036 1162 raft_consensus.cc:383] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.482120 1162 raft_consensus.cc:738] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.482506 1165 ts_tablet_manager.cc:1918] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.482563 1162 consensus_queue.cc:260] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.482836 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c1f78195e62c4c148f6b298e2cbc055e" } modify_external_catalogs: true
I20250115 14:51:51.483008 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c1f78195e62c4c148f6b298e2cbc055e" } modify_external_catalogs: true
I20250115 14:51:51.483157 1162 ts_tablet_manager.cc:1428] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:51.483397 1162 tablet_bootstrap.cc:492] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.482622 1165 log.cc:1198] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/c5030b357a804d70b13a93b1d5d5fed1
I20250115 14:51:51.483929 1165 ts_tablet_manager.cc:1939] T c5030b357a804d70b13a93b1d5d5fed1 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:51.484962 1162 tablet_bootstrap.cc:654] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.489686 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet c5030b357a804d70b13a93b1d5d5fed1 (table default.Table_05cc51b3f5ba4d518525c47cbfba4527 [id=7fc50307921343ddb6368d736c8adbfc]) successfully deleted
W20250115 14:51:51.489956 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 3ef5e4d75580465c9c0e36135503f332 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef5e4d75580465c9c0e36135503f332 already in progress: creating tablet
I20250115 14:51:51.490612 890 tablet_service.cc:1514] Processing DeleteTablet for tablet c5030b357a804d70b13a93b1d5d5fed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.490806 1167 raft_consensus.cc:2238] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:51.491595 1167 pending_rounds.cc:62] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:51.491823 1167 pending_rounds.cc:69] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558306192171008 op_type: NO_OP noop_request { }
I20250115 14:51:51.491999 1167 raft_consensus.cc:2883] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:51.492125 1167 raft_consensus.cc:2267] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.492487 1169 raft_consensus.cc:491] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:51.492627 1169 raft_consensus.cc:513] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:51.494058 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f224f857b1644596853acd9e3a2a4a5f has no consensus state
I20250115 14:51:51.494210 1165 tablet_replica.cc:331] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:51.494385 1169 leader_election.cc:290] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:51.494946 1168 tablet_bootstrap.cc:492] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.495208 1168 ts_tablet_manager.cc:1397] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.001s
I20250115 14:51:51.496084 1168 raft_consensus.cc:357] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.496266 1168 raft_consensus.cc:383] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.496372 1168 raft_consensus.cc:738] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.496577 1168 consensus_queue.cc:260] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.497726 1168 ts_tablet_manager.cc:1428] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:51.498261 1168 tablet_bootstrap.cc:492] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.500033 889 tablet_service.cc:1467] Processing CreateTablet for tablet c322301320ba425a94d7321c08f7f9e6 (DEFAULT_TABLE table=default.Table_dd152d863f394374a58bf845d3d259e2 [id=d5216eacff2b46ad96642b0ef7fd6403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.500469 1167 ts_tablet_manager.cc:1905] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.502517 1168 tablet_bootstrap.cc:654] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.502732 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c322301320ba425a94d7321c08f7f9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.509137 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 7bdb617ac1c042d282873f2c01bf6162 (table default.Table_7de6e49996a0432d8f8bfa56c51d0782 [id=ff5b7f4d675f496eaf7dd60b0cca7435]) successfully deleted
I20250115 14:51:51.509300 1167 ts_tablet_manager.cc:1918] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:51.509415 1167 log.cc:1198] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f1328f038ad3423a8f0edcb13e0f555a
I20250115 14:51:51.509840 1167 ts_tablet_manager.cc:1939] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.510519 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f1328f038ad3423a8f0edcb13e0f555a (table default.Table_8d7e6f90e6a64f5d84f728b16b2deea8 [id=d7ba4cda40964e1d839efe9e0f1615c0]) successfully deleted
I20250115 14:51:51.511725 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd6d837c64ad4503a224ced67ff1d053. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:51.512812 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet be18f1b6d8a24fabbe818811ef646c62 because the tablet was not found. No further retry: Not found: Tablet not found: be18f1b6d8a24fabbe818811ef646c62
W20250115 14:51:51.513208 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2b4259dfb8ee4b618c0bacc910addd0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b4259dfb8ee4b618c0bacc910addd0a already in progress: creating tablet
I20250115 14:51:51.513363 1167 tablet_replica.cc:331] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.513577 1167 raft_consensus.cc:2238] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.513723 1167 raft_consensus.cc:2267] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.513700 1165 raft_consensus.cc:2238] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.513890 1165 raft_consensus.cc:2267] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.514525 1167 ts_tablet_manager.cc:1905] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.517393 1165 ts_tablet_manager.cc:1905] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.517705 756 tablet_service.cc:1467] Processing CreateTablet for tablet 22edaa6e70d942ba98777973476f449d (DEFAULT_TABLE table=default.Table_8d7e6f90e6a64f5d84f728b16b2deea8 [id=d7ba4cda40964e1d839efe9e0f1615c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.518373 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c322301320ba425a94d7321c08f7f9e6 (DEFAULT_TABLE table=default.Table_dd152d863f394374a58bf845d3d259e2 [id=d5216eacff2b46ad96642b0ef7fd6403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.518769 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c322301320ba425a94d7321c08f7f9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.519061 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 22edaa6e70d942ba98777973476f449d (DEFAULT_TABLE table=default.Table_8d7e6f90e6a64f5d84f728b16b2deea8 [id=d7ba4cda40964e1d839efe9e0f1615c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.521572 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 85ab1ccf420d41bd98cfda29caf5a619 (table default.Table_575e630b7f8d4823bfa54f3ed4a488fb [id=af4d0a3d017c4e2cbc04fa416fa68e5a]). Expected version 1 got 0
I20250115 14:51:51.521900 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22edaa6e70d942ba98777973476f449d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.522766 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 85ab1ccf420d41bd98cfda29caf5a619 (table default.Table_575e630b7f8d4823bfa54f3ed4a488fb [id=af4d0a3d017c4e2cbc04fa416fa68e5a]). Expected version 1 got 0
I20250115 14:51:51.523239 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22edaa6e70d942ba98777973476f449d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:51.523910 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85ab1ccf420d41bd98cfda29caf5a619 consensus state has no leader
I20250115 14:51:51.523854 754 tablet_service.cc:1467] Processing CreateTablet for tablet c322301320ba425a94d7321c08f7f9e6 (DEFAULT_TABLE table=default.Table_dd152d863f394374a58bf845d3d259e2 [id=d5216eacff2b46ad96642b0ef7fd6403]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.529345 1025 tablet_service.cc:1467] Processing CreateTablet for tablet d6a9fb932c5b4545a4609d568f0e5494 (DEFAULT_TABLE table=default.Table_65ba1d73ba754551a967823306bb4d16 [id=3f3fecceceb3464bb12462bb3041b458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.529660 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c322301320ba425a94d7321c08f7f9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.529726 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6a9fb932c5b4545a4609d568f0e5494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.532418 1164 tablet_bootstrap.cc:492] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:51.532564 1024 tablet_service.cc:1467] Processing CreateTablet for tablet e9e5bc6131944fcab9c550caac4f8d85 (DEFAULT_TABLE table=default.Table_3529669cc14746fda74ad45528cbcf4c [id=33e3df28f2d445c5ba4e8b0ce4458c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.532884 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e5bc6131944fcab9c550caac4f8d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.533182 1166 tablet_replica.cc:331] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:51.535506 755 tablet_service.cc:1467] Processing CreateTablet for tablet e9e5bc6131944fcab9c550caac4f8d85 (DEFAULT_TABLE table=default.Table_3529669cc14746fda74ad45528cbcf4c [id=33e3df28f2d445c5ba4e8b0ce4458c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.536562 1023 tablet_service.cc:1467] Processing CreateTablet for tablet f224f857b1644596853acd9e3a2a4a5f (DEFAULT_TABLE table=default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.536871 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f224f857b1644596853acd9e3a2a4a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.538678 1166 raft_consensus.cc:2238] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.540575 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 74bfda3861d148bc8b67ea071909d22d (DEFAULT_TABLE table=default.Table_a78c10f9e6384d7d89d3add6f1371429 [id=2c59ffcdee944c4289ad5dd38dc2e8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.540884 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74bfda3861d148bc8b67ea071909d22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.541721 756 tablet_service.cc:1467] Processing CreateTablet for tablet f224f857b1644596853acd9e3a2a4a5f (DEFAULT_TABLE table=default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.542095 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 3a38a6bc32314396b497e7ae166a63f7 (DEFAULT_TABLE table=default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.542172 754 tablet_service.cc:1467] Processing CreateTablet for tablet 74bfda3861d148bc8b67ea071909d22d (DEFAULT_TABLE table=default.Table_a78c10f9e6384d7d89d3add6f1371429 [id=2c59ffcdee944c4289ad5dd38dc2e8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.542377 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a38a6bc32314396b497e7ae166a63f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.543478 1168 tablet_bootstrap.cc:492] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.543642 1168 ts_tablet_manager.cc:1397] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.004s
I20250115 14:51:51.544386 1168 raft_consensus.cc:357] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.544598 1168 raft_consensus.cc:383] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.544656 1168 raft_consensus.cc:738] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.544620 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 13c00f03edc74af8b084bb236487a347 (DEFAULT_TABLE table=default.Table_64abfdac91a54d6bb59ccfc7791a170c [id=358f8fad6d944439847ea4ce82f1213a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.544888 1168 consensus_queue.cc:260] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.545763 888 tablet_service.cc:1467] Processing CreateTablet for tablet e9e5bc6131944fcab9c550caac4f8d85 (DEFAULT_TABLE table=default.Table_3529669cc14746fda74ad45528cbcf4c [id=33e3df28f2d445c5ba4e8b0ce4458c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.545950 892 tablet_service.cc:1467] Processing CreateTablet for tablet 74bfda3861d148bc8b67ea071909d22d (DEFAULT_TABLE table=default.Table_a78c10f9e6384d7d89d3add6f1371429 [id=2c59ffcdee944c4289ad5dd38dc2e8cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.546155 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e5bc6131944fcab9c550caac4f8d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.546116 890 tablet_service.cc:1467] Processing CreateTablet for tablet f224f857b1644596853acd9e3a2a4a5f (DEFAULT_TABLE table=default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.546353 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74bfda3861d148bc8b67ea071909d22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.546555 893 tablet_service.cc:1467] Processing CreateTablet for tablet 13c00f03edc74af8b084bb236487a347 (DEFAULT_TABLE table=default.Table_64abfdac91a54d6bb59ccfc7791a170c [id=358f8fad6d944439847ea4ce82f1213a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.546891 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13c00f03edc74af8b084bb236487a347. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.547216 891 tablet_service.cc:1467] Processing CreateTablet for tablet 8d43cd66b18f4649894a629637037822 (DEFAULT_TABLE table=default.Table_733144094e85412998e706e7a0f93dbf [id=e1e34c5d139f4810a8f45a223e459e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.547587 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d43cd66b18f4649894a629637037822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.550704 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3a38a6bc32314396b497e7ae166a63f7 (DEFAULT_TABLE table=default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.551065 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a38a6bc32314396b497e7ae166a63f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.551826 883 tablet_service.cc:1467] Processing CreateTablet for tablet 6326b1526191464485855b657e34a2b9 (DEFAULT_TABLE table=default.Table_662e371503c24e6cacf91cb2afe19d53 [id=ade9599dbabb4c298debe2172859adcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.552395 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6326b1526191464485855b657e34a2b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.552773 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f224f857b1644596853acd9e3a2a4a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.555646 1168 ts_tablet_manager.cc:1428] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250115 14:51:51.555774 885 tablet_service.cc:1467] Processing CreateTablet for tablet 5d51e53b4ffa45bfbb9553d7a5870844 (DEFAULT_TABLE table=default.Table_8003836a7a5c4f79a7bec455f2d8a325 [id=817d675e581a4abf848167ce48e46bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.556166 884 tablet_service.cc:1467] Processing CreateTablet for tablet bf909c84eb314d95b73b6de82aebe9f2 (DEFAULT_TABLE table=default.Table_319faebd004b4a1e93821d947ef288d0 [id=55e89953f6914380bb5b2d81aa9da51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.556615 1167 ts_tablet_manager.cc:1918] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.556845 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 649504c31def4f3f9c30879033b58fbe
I20250115 14:51:51.560923 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 85ab1ccf420d41bd98cfda29caf5a619 (table default.Table_eb8583ed6e0c445b9883e8391d53b878 [id=af4d0a3d017c4e2cbc04fa416fa68e5a]). Expected version 2 got 0
I20250115 14:51:51.561353 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a78c10f9e6384d7d89d3add6f1371429" } modify_external_catalogs: true
I20250115 14:51:51.561537 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a78c10f9e6384d7d89d3add6f1371429" } modify_external_catalogs: true
I20250115 14:51:51.564170 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 6326b1526191464485855b657e34a2b9 (DEFAULT_TABLE table=default.Table_662e371503c24e6cacf91cb2afe19d53 [id=ade9599dbabb4c298debe2172859adcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.564540 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 8d43cd66b18f4649894a629637037822 (DEFAULT_TABLE table=default.Table_733144094e85412998e706e7a0f93dbf [id=e1e34c5d139f4810a8f45a223e459e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.564925 1018 tablet_service.cc:1467] Processing CreateTablet for tablet bf909c84eb314d95b73b6de82aebe9f2 (DEFAULT_TABLE table=default.Table_319faebd004b4a1e93821d947ef288d0 [id=55e89953f6914380bb5b2d81aa9da51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.565277 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 5d51e53b4ffa45bfbb9553d7a5870844 (DEFAULT_TABLE table=default.Table_8003836a7a5c4f79a7bec455f2d8a325 [id=817d675e581a4abf848167ce48e46bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.565723 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 649504c31def4f3f9c30879033b58fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.565852 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e20ad209d7044b09935691f002b9998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.566015 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0d566e954045fc85c4fbfe8052b050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.566395 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 649504c31def4f3f9c30879033b58fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.566982 748 tablet_service.cc:1467] Processing CreateTablet for tablet 6326b1526191464485855b657e34a2b9 (DEFAULT_TABLE table=default.Table_662e371503c24e6cacf91cb2afe19d53 [id=ade9599dbabb4c298debe2172859adcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.569223 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 715274a681f447c594e8fcd38d6fd609
I20250115 14:51:51.569868 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f8346c87744f46daa623c414e7307299
I20250115 14:51:51.569869 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9bbd41d946d245b78e6686f8fae55449
I20250115 14:51:51.571554 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet f8346c87744f46daa623c414e7307299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.571635 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbd41d946d245b78e6686f8fae55449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.571827 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 715274a681f447c594e8fcd38d6fd609 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.573786 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 715274a681f447c594e8fcd38d6fd609 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.573971 882 tablet_service.cc:1514] Processing DeleteTablet for tablet f8346c87744f46daa623c414e7307299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.574110 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbd41d946d245b78e6686f8fae55449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.574223 1167 log.cc:1198] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/101087eee11441f299d5b75e5c80b66b
I20250115 14:51:51.574591 1167 ts_tablet_manager.cc:1939] T 101087eee11441f299d5b75e5c80b66b P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.580657 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 74bfda3861d148bc8b67ea071909d22d
I20250115 14:51:51.584357 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d51e53b4ffa45bfbb9553d7a5870844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.587898 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 59f25f2f228143c98f6cd6ccb0cf99d7 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.588624 1168 tablet_bootstrap.cc:492] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.590122 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 715274a681f447c594e8fcd38d6fd609 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.590431 1168 tablet_bootstrap.cc:654] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.591498 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 74bfda3861d148bc8b67ea071909d22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.591964 889 tablet_service.cc:1467] Processing CreateTablet for tablet d6a9fb932c5b4545a4609d568f0e5494 (DEFAULT_TABLE table=default.Table_65ba1d73ba754551a967823306bb4d16 [id=3f3fecceceb3464bb12462bb3041b458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.592708 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf909c84eb314d95b73b6de82aebe9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.599596 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e20ad209d7044b09935691f002b9998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.600095 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 74bfda3861d148bc8b67ea071909d22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.600461 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 9bbd41d946d245b78e6686f8fae55449 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.600629 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6a9fb932c5b4545a4609d568f0e5494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.601150 1167 tablet_replica.cc:331] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.601141 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 718f5f0300114292bf838f5f988ab566 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.601352 1167 raft_consensus.cc:2238] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.601533 1167 raft_consensus.cc:2267] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.602154 1167 ts_tablet_manager.cc:1905] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.603813 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 101087eee11441f299d5b75e5c80b66b (table default.Table_f04c4734007f45f2b1b6dbe3cfeec857 [id=05c9f91a89fb43c8918c6cfe74360e43]) successfully deleted
W20250115 14:51:51.604080 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2b4259dfb8ee4b618c0bacc910addd0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b4259dfb8ee4b618c0bacc910addd0a already in progress: creating tablet
I20250115 14:51:51.605070 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c68a808d4c4744fe92a97753d517180a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.605666 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c68a808d4c4744fe92a97753d517180a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.607703 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9e5bc6131944fcab9c550caac4f8d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.610235 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6326b1526191464485855b657e34a2b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.610298 1168 tablet_bootstrap.cc:492] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.610430 1168 ts_tablet_manager.cc:1397] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.012s
I20250115 14:51:51.608109 1167 ts_tablet_manager.cc:1918] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.610646 1167 log.cc:1198] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/c5030b357a804d70b13a93b1d5d5fed1
I20250115 14:51:51.611016 1167 ts_tablet_manager.cc:1939] T c5030b357a804d70b13a93b1d5d5fed1 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.611227 1168 raft_consensus.cc:357] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.611407 1168 raft_consensus.cc:383] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.611490 1168 raft_consensus.cc:738] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.611780 1167 tablet_replica.cc:331] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.611668 1168 consensus_queue.cc:260] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.611950 1167 raft_consensus.cc:2238] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.612090 1167 raft_consensus.cc:2267] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.613620 1168 ts_tablet_manager.cc:1428] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:51.613893 1168 tablet_bootstrap.cc:492] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.615224 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet c5030b357a804d70b13a93b1d5d5fed1 (table default.Table_05cc51b3f5ba4d518525c47cbfba4527 [id=7fc50307921343ddb6368d736c8adbfc]) successfully deleted
I20250115 14:51:51.616400 751 tablet_service.cc:1467] Processing CreateTablet for tablet 3a38a6bc32314396b497e7ae166a63f7 (DEFAULT_TABLE table=default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.616643 1168 tablet_bootstrap.cc:654] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.616751 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a38a6bc32314396b497e7ae166a63f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.617858 1167 ts_tablet_manager.cc:1905] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.621428 749 tablet_service.cc:1467] Processing CreateTablet for tablet 13c00f03edc74af8b084bb236487a347 (DEFAULT_TABLE table=default.Table_64abfdac91a54d6bb59ccfc7791a170c [id=358f8fad6d944439847ea4ce82f1213a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.621786 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13c00f03edc74af8b084bb236487a347. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.625165 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_735fcf813ef3439fa578a7fae3066b42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.625701 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_735fcf813ef3439fa578a7fae3066b42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.626945 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13c00f03edc74af8b084bb236487a347. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.626916 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_de5e692c0fa544a5975b1cc474c6a6d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.627372 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de5e692c0fa544a5975b1cc474c6a6d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.628466 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bbf01a6c24b9449b8da3ee9428b7bd98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.628902 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbf01a6c24b9449b8da3ee9428b7bd98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.629297 1168 tablet_bootstrap.cc:492] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.629458 1168 ts_tablet_manager.cc:1397] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.001s
I20250115 14:51:51.629812 752 tablet_service.cc:1467] Processing CreateTablet for tablet d6a9fb932c5b4545a4609d568f0e5494 (DEFAULT_TABLE table=default.Table_65ba1d73ba754551a967823306bb4d16 [id=3f3fecceceb3464bb12462bb3041b458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.630138 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6326b1526191464485855b657e34a2b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.630182 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6a9fb932c5b4545a4609d568f0e5494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.631058 1168 raft_consensus.cc:357] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.632372 747 tablet_service.cc:1467] Processing CreateTablet for tablet 5d51e53b4ffa45bfbb9553d7a5870844 (DEFAULT_TABLE table=default.Table_8003836a7a5c4f79a7bec455f2d8a325 [id=817d675e581a4abf848167ce48e46bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.632553 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d43cd66b18f4649894a629637037822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.632828 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d51e53b4ffa45bfbb9553d7a5870844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.633173 1167 ts_tablet_manager.cc:1918] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.633294 1167 log.cc:1198] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/649504c31def4f3f9c30879033b58fbe
I20250115 14:51:51.633611 1167 ts_tablet_manager.cc:1939] T 649504c31def4f3f9c30879033b58fbe P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.634881 1167 tablet_replica.cc:331] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.635046 1167 raft_consensus.cc:2238] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.635196 1167 raft_consensus.cc:2267] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.635509 1168 raft_consensus.cc:383] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.635576 1168 raft_consensus.cc:738] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.635797 1168 consensus_queue.cc:260] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.636379 746 tablet_service.cc:1467] Processing CreateTablet for tablet bf909c84eb314d95b73b6de82aebe9f2 (DEFAULT_TABLE table=default.Table_319faebd004b4a1e93821d947ef288d0 [id=55e89953f6914380bb5b2d81aa9da51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.636510 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf909c84eb314d95b73b6de82aebe9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.636817 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf909c84eb314d95b73b6de82aebe9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.637728 1168 ts_tablet_manager.cc:1428] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:51:51.638015 1168 tablet_bootstrap.cc:492] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.639575 1168 tablet_bootstrap.cc:654] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.640089 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task__Z8ae/build/asan/bin/kudu
/tmp/dist-test-task__Z8ae/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40227
--webserver_interface=127.0.104.60
--webserver_port=37445
--builtin_ntp_servers=127.0.104.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250115 14:51:51.644369 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 649504c31def4f3f9c30879033b58fbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.644519 1168 tablet_bootstrap.cc:492] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.644567 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 715274a681f447c594e8fcd38d6fd609 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.644639 1168 ts_tablet_manager.cc:1397] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20250115 14:51:51.644711 743 tablet_service.cc:1514] Processing DeleteTablet for tablet f8346c87744f46daa623c414e7307299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.644866 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbd41d946d245b78e6686f8fae55449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.645363 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d51e53b4ffa45bfbb9553d7a5870844. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.645327 1168 raft_consensus.cc:357] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.645516 1168 raft_consensus.cc:383] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.645589 1168 raft_consensus.cc:738] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.646374 1167 ts_tablet_manager.cc:1905] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.650717 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3e3a408924cc4cb0b5a408089e63bb1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.651221 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e3a408924cc4cb0b5a408089e63bb1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.668895 1168 consensus_queue.cc:260] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.669143 1174 raft_consensus.cc:491] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:51.669278 1174 raft_consensus.cc:513] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.669909 1174 leader_election.cc:290] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:51.669911 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 649504c31def4f3f9c30879033b58fbe (table default.Table_c1f78195e62c4c148f6b298e2cbc055e [id=f6f31e321abb451c877540a31b2fb976]) successfully deleted
W20250115 14:51:51.670151 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 715274a681f447c594e8fcd38d6fd609 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 715274a681f447c594e8fcd38d6fd609 already in progress: creating tablet
W20250115 14:51:51.670331 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f8346c87744f46daa623c414e7307299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8346c87744f46daa623c414e7307299 already in progress: creating tablet
W20250115 14:51:51.670539 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9bbd41d946d245b78e6686f8fae55449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bbd41d946d245b78e6686f8fae55449 already in progress: creating tablet
I20250115 14:51:51.670504 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718f5f0300114292bf838f5f988ab566" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:51.671232 847 leader_election.cc:343] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.672734 1168 ts_tablet_manager.cc:1428] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250115 14:51:51.673025 1168 tablet_bootstrap.cc:492] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.674706 1168 tablet_bootstrap.cc:654] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.677886 1166 raft_consensus.cc:2267] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.678632 1166 ts_tablet_manager.cc:1905] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.680151 1167 ts_tablet_manager.cc:1918] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.681293 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f224f857b1644596853acd9e3a2a4a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.684419 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74bfda3861d148bc8b67ea071909d22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.680284 1167 log.cc:1198] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/0e20ad209d7044b09935691f002b9998
I20250115 14:51:51.689236 1167 ts_tablet_manager.cc:1939] T 0e20ad209d7044b09935691f002b9998 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.688169 1168 tablet_bootstrap.cc:492] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.690315 1168 ts_tablet_manager.cc:1397] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20250115 14:51:51.690696 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 0e20ad209d7044b09935691f002b9998 (table default.Table_5d2e74e77f87456396aed66d4d21f840 [id=1fc8e792fbbe4faf85699a5a99b99908]) successfully deleted
W20250115 14:51:51.690971 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 74bfda3861d148bc8b67ea071909d22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74bfda3861d148bc8b67ea071909d22d already in progress: creating tablet
I20250115 14:51:51.691187 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1328f038ad3423a8f0edcb13e0f555a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:51.691655 1168 raft_consensus.cc:357] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.691896 1168 raft_consensus.cc:383] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.691972 1168 raft_consensus.cc:738] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.691918 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1328f038ad3423a8f0edcb13e0f555a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:51.692267 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 74bfda3861d148bc8b67ea071909d22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.692901 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_68b340d318b14c078d634b22c471d979"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:51.693120 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d4858b82cd50473bbd6fda2dcf3bd299" } new_table_name: "default.Table_fb7d4709dd544378b8802e3294d6e441" modify_external_catalogs: true
I20250115 14:51:51.693176 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_02165515be8347129d74a22a54a84ad9" } new_table_name: "default.Table_47396588856f48a6ae9cb4d6469a39d1" modify_external_catalogs: true
W20250115 14:51:51.693506 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68b340d318b14c078d634b22c471d979 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.694535 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8d37d2aba1094de3a9bc2eabee94643d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:51.694800 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8d7e6f90e6a64f5d84f728b16b2deea8" } new_table_name: "default.Table_1b3a658f6eda417cb956eba9cdc5a2af" modify_external_catalogs: true
W20250115 14:51:51.695050 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d37d2aba1094de3a9bc2eabee94643d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.696657 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8d43cd66b18f4649894a629637037822 (DEFAULT_TABLE table=default.Table_733144094e85412998e706e7a0f93dbf [id=e1e34c5d139f4810a8f45a223e459e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.697053 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d43cd66b18f4649894a629637037822. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.697284 1168 consensus_queue.cc:260] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.698858 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 8d43cd66b18f4649894a629637037822 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.699182 1169 raft_consensus.cc:491] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:51.699352 1169 raft_consensus.cc:513] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.699946 1169 leader_election.cc:290] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127)
I20250115 14:51:51.705507 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef5e4d75580465c9c0e36135503f332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:51.706192 848 leader_election.cc:343] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:51.706717 848 leader_election.cc:343] T f1328f038ad3423a8f0edcb13e0f555a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.707207 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718f5f0300114292bf838f5f988ab566" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:51.707902 848 leader_election.cc:343] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.708045 848 leader_election.cc:304] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:51.708343 1174 raft_consensus.cc:2743] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:51.708806 1167 tablet_replica.cc:331] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.709000 1167 raft_consensus.cc:2238] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.709160 1167 raft_consensus.cc:2267] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.709628 1168 ts_tablet_manager.cc:1428] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250115 14:51:51.711377 1167 ts_tablet_manager.cc:1905] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.712206 1168 tablet_bootstrap.cc:492] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.713395 1164 ts_tablet_manager.cc:1397] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.092s user 0.017s sys 0.025s
I20250115 14:51:51.714186 1164 raft_consensus.cc:357] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.714380 1164 raft_consensus.cc:383] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.714444 1164 raft_consensus.cc:738] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.715186 1168 tablet_bootstrap.cc:654] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.719449 1168 tablet_bootstrap.cc:492] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.719609 1168 ts_tablet_manager.cc:1397] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250115 14:51:51.720435 1168 raft_consensus.cc:357] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.720623 1168 raft_consensus.cc:383] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.720721 1168 raft_consensus.cc:738] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.720353 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0d566e954045fc85c4fbfe8052b050" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:51.720966 912 raft_consensus.cc:2463] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
W20250115 14:51:51.721525 983 leader_election.cc:343] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 649504c31def4f3f9c30879033b58fbe
I20250115 14:51:51.722036 983 leader_election.cc:304] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:51.722505 1169 raft_consensus.cc:2798] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:51.722599 1169 raft_consensus.cc:491] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:51.722688 1169 raft_consensus.cc:3054] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:51.723033 1168 consensus_queue.cc:260] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.725301 1169 raft_consensus.cc:513] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.726029 1169 leader_election.cc:290] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127)
I20250115 14:51:51.726375 1162 tablet_bootstrap.cc:492] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.726531 1162 ts_tablet_manager.cc:1397] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.243s user 0.018s sys 0.041s
I20250115 14:51:51.726663 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0d566e954045fc85c4fbfe8052b050" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:51.726827 912 raft_consensus.cc:3054] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:51.727291 1162 raft_consensus.cc:357] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.727494 1162 raft_consensus.cc:383] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.727563 1162 raft_consensus.cc:738] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.727826 1162 consensus_queue.cc:260] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.729265 1162 ts_tablet_manager.cc:1428] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:51.729494 912 raft_consensus.cc:2463] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:51.729668 1168 ts_tablet_manager.cc:1428] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250115 14:51:51.729864 983 leader_election.cc:304] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:51.730017 1162 tablet_bootstrap.cc:492] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.730123 1169 raft_consensus.cc:2798] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:51.730211 1164 consensus_queue.cc:260] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.735329 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0d566e954045fc85c4fbfe8052b050" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:51.735910 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0d566e954045fc85c4fbfe8052b050" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:51.736263 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649504c31def4f3f9c30879033b58fbe" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:51.736562 775 raft_consensus.cc:2463] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
W20250115 14:51:51.739199 984 leader_election.cc:343] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.739564 1167 ts_tablet_manager.cc:1918] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:51.739655 984 leader_election.cc:343] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.739698 1167 log.cc:1198] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/3ef5e4d75580465c9c0e36135503f332
I20250115 14:51:51.739933 984 leader_election.cc:304] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d
I20250115 14:51:51.740042 1167 ts_tablet_manager.cc:1939] T 3ef5e4d75580465c9c0e36135503f332 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.740356 1168 tablet_bootstrap.cc:492] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.741153 1164 ts_tablet_manager.cc:1428] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250115 14:51:51.741109 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 3ef5e4d75580465c9c0e36135503f332 (table default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1 [id=546100c082df4fb0984876cf4c60009c]) successfully deleted
I20250115 14:51:51.741437 1169 raft_consensus.cc:695] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:51.741854 1169 consensus_queue.cc:237] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.744069 1164 tablet_bootstrap.cc:492] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:51.745388 1162 tablet_bootstrap.cc:654] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.745558 1164 tablet_bootstrap.cc:654] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.747730 1168 tablet_bootstrap.cc:654] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.748323 1165 ts_tablet_manager.cc:1918] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.748466 1165 log.cc:1198] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/65f2f961e08846d09df469b74abd5da1
I20250115 14:51:51.748811 1165 ts_tablet_manager.cc:1939] T 65f2f961e08846d09df469b74abd5da1 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:51.749264 1225 raft_consensus.cc:491] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:51.749301 1166 ts_tablet_manager.cc:1918] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.749409 1166 log.cc:1198] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f1328f038ad3423a8f0edcb13e0f555a
I20250115 14:51:51.749411 1225 raft_consensus.cc:513] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.749737 1166 ts_tablet_manager.cc:1939] T f1328f038ad3423a8f0edcb13e0f555a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:51.750551 1231 raft_consensus.cc:2798] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:51.750573 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f1328f038ad3423a8f0edcb13e0f555a (table default.Table_8d7e6f90e6a64f5d84f728b16b2deea8 [id=d7ba4cda40964e1d839efe9e0f1615c0]) successfully deleted
I20250115 14:51:51.750656 1231 raft_consensus.cc:491] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:51.750665 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718f5f0300114292bf838f5f988ab566" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:51.750771 1231 raft_consensus.cc:3054] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:51.750862 912 raft_consensus.cc:2463] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:51.751248 983 leader_election.cc:304] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:51.751557 1225 leader_election.cc:290] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:51.751739 1225 raft_consensus.cc:2798] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:51.751838 1225 raft_consensus.cc:491] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:51.751922 1225 raft_consensus.cc:3054] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:51.753311 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_22fff0dd323f40378324128fc8bdddd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.753849 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22fff0dd323f40378324128fc8bdddd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.754587 1225 raft_consensus.cc:513] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.755456 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718f5f0300114292bf838f5f988ab566" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:51.755638 912 raft_consensus.cc:3054] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:51.757051 1168 tablet_bootstrap.cc:492] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.757210 1168 ts_tablet_manager.cc:1397] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20250115 14:51:51.757938 1168 raft_consensus.cc:357] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.758150 1168 raft_consensus.cc:383] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.758208 912 raft_consensus.cc:2463] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:51.758527 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718f5f0300114292bf838f5f988ab566" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:51.758595 983 leader_election.cc:304] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:51.758883 1225 leader_election.cc:290] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:51.759063 1225 raft_consensus.cc:2798] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:51.759179 1225 raft_consensus.cc:695] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:51.760041 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_49eb379e719f4677b13269d15ef7efad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:51.760437 1162 tablet_bootstrap.cc:492] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
W20250115 14:51:51.760532 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49eb379e719f4677b13269d15ef7efad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.760572 1162 ts_tablet_manager.cc:1397] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.002s
I20250115 14:51:51.760905 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718f5f0300114292bf838f5f988ab566" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:51.758226 1168 raft_consensus.cc:738] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.759449 1225 consensus_queue.cc:237] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.761821 1166 tablet_replica.cc:331] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:51.762005 1166 raft_consensus.cc:2238] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.762115 1168 consensus_queue.cc:260] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.762248 1166 raft_consensus.cc:2267] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.762781 1166 ts_tablet_manager.cc:1905] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:51.764840 984 leader_election.cc:343] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:51.765276 984 leader_election.cc:343] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.765966 1168 ts_tablet_manager.cc:1428] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250115 14:51:51.767074 1164 tablet_bootstrap.cc:492] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:51.767194 1164 ts_tablet_manager.cc:1397] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20250115 14:51:51.767865 1164 raft_consensus.cc:357] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.768033 1164 raft_consensus.cc:383] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.768102 1164 raft_consensus.cc:738] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.768038 1231 raft_consensus.cc:513] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:51.768388 1164 consensus_queue.cc:260] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.768805 1168 tablet_bootstrap.cc:492] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.769027 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "649504c31def4f3f9c30879033b58fbe" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:51.769327 983 leader_election.cc:343] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 649504c31def4f3f9c30879033b58fbe
W20250115 14:51:51.769501 984 leader_election.cc:343] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: Tablet not RUNNING: STOPPED
I20250115 14:51:51.769644 984 leader_election.cc:304] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:51.769927 1226 raft_consensus.cc:2743] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250115 14:51:51.770032 1162 raft_consensus.cc:357] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.770206 1162 raft_consensus.cc:383] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.770283 1162 raft_consensus.cc:738] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.770483 1162 consensus_queue.cc:260] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.770916 1162 ts_tablet_manager.cc:1428] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250115 14:51:51.771186 1162 tablet_bootstrap.cc:492] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.771883 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f2f961e08846d09df469b74abd5da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.772732 1162 tablet_bootstrap.cc:654] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.773247 1168 tablet_bootstrap.cc:654] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.773598 1164 ts_tablet_manager.cc:1428] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250115 14:51:51.773792 1231 leader_election.cc:290] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:51.777631 1164 tablet_bootstrap.cc:492] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:51.779263 1164 tablet_bootstrap.cc:654] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.781450 1162 tablet_bootstrap.cc:492] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.781602 1162 ts_tablet_manager.cc:1397] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250115 14:51:51.782315 1162 raft_consensus.cc:357] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.782537 1162 raft_consensus.cc:383] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.782600 1162 raft_consensus.cc:738] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.783741 1166 ts_tablet_manager.cc:1918] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.783860 1166 log.cc:1198] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/649504c31def4f3f9c30879033b58fbe
I20250115 14:51:51.784188 1166 ts_tablet_manager.cc:1939] T 649504c31def4f3f9c30879033b58fbe P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:51.785352 1027 tablet_service.cc:1467] Processing CreateTablet for tablet f6205e20a1984ccabccc45b93f9c7fa8 (DEFAULT_TABLE table=default.Table_c68a808d4c4744fe92a97753d517180a [id=7222964ef7514b2ba94aa1527810947a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.785498 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.785776 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6205e20a1984ccabccc45b93f9c7fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.785768 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0d566e954045fc85c4fbfe8052b050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.786183 884 tablet_service.cc:1467] Processing CreateTablet for tablet f6205e20a1984ccabccc45b93f9c7fa8 (DEFAULT_TABLE table=default.Table_c68a808d4c4744fe92a97753d517180a [id=7222964ef7514b2ba94aa1527810947a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.786521 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6205e20a1984ccabccc45b93f9c7fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.788558 1164 tablet_bootstrap.cc:492] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:51.788720 1164 ts_tablet_manager.cc:1397] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250115 14:51:51.788715 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 718f5f0300114292bf838f5f988ab566 with delete_type TABLET_DATA_DELETED (replaced by tablet be18f1b6d8a24fabbe818811ef646c62) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.789355 1164 raft_consensus.cc:357] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.789553 1164 raft_consensus.cc:383] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.789624 1164 raft_consensus.cc:738] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.789850 1164 consensus_queue.cc:260] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.789938 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 649504c31def4f3f9c30879033b58fbe (table default.Table_c1f78195e62c4c148f6b298e2cbc055e [id=f6f31e321abb451c877540a31b2fb976]) successfully deleted
W20250115 14:51:51.790169 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 715274a681f447c594e8fcd38d6fd609 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 715274a681f447c594e8fcd38d6fd609 already in progress: creating tablet
W20250115 14:51:51.790383 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f8346c87744f46daa623c414e7307299 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8346c87744f46daa623c414e7307299 already in progress: creating tablet
W20250115 14:51:51.790567 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9bbd41d946d245b78e6686f8fae55449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bbd41d946d245b78e6686f8fae55449 already in progress: creating tablet
W20250115 14:51:51.790753 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 74bfda3861d148bc8b67ea071909d22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74bfda3861d148bc8b67ea071909d22d already in progress: creating tablet
W20250115 14:51:51.790964 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 85ab1ccf420d41bd98cfda29caf5a619 (table default.Table_eb8583ed6e0c445b9883e8391d53b878 [id=af4d0a3d017c4e2cbc04fa416fa68e5a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 85ab1ccf420d41bd98cfda29caf5a619 consensus state has no leader
I20250115 14:51:51.792966 754 tablet_service.cc:1467] Processing CreateTablet for tablet f6205e20a1984ccabccc45b93f9c7fa8 (DEFAULT_TABLE table=default.Table_c68a808d4c4744fe92a97753d517180a [id=7222964ef7514b2ba94aa1527810947a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.793313 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6205e20a1984ccabccc45b93f9c7fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.793610 1162 consensus_queue.cc:260] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.794890 1162 ts_tablet_manager.cc:1428] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250115 14:51:51.795184 1162 tablet_bootstrap.cc:492] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.796782 1162 tablet_bootstrap.cc:654] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.803854 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 510ac6d6ee3e488e8b20f3e39a44ab9a with delete_type TABLET_DATA_DELETED (replaced by tablet bd6d837c64ad4503a224ced67ff1d053) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.805806 1168 tablet_bootstrap.cc:492] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.805961 1168 ts_tablet_manager.cc:1397] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.024s
I20250115 14:51:51.806782 1168 raft_consensus.cc:357] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.806969 1168 raft_consensus.cc:383] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.807073 1168 raft_consensus.cc:738] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.807281 1168 consensus_queue.cc:260] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:51.807592 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2529350d9264a1793c605ea9d5df9c6 consensus state has no leader
I20250115 14:51:51.807842 1168 ts_tablet_manager.cc:1428] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:51.807960 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_65ba1d73ba754551a967823306bb4d16" } new_table_name: "default.Table_858dba41222b48a794cf0888977761c8" modify_external_catalogs: true
I20250115 14:51:51.808123 1168 tablet_bootstrap.cc:492] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.810097 1168 tablet_bootstrap.cc:654] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:51.810297 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22edaa6e70d942ba98777973476f449d consensus state has no leader
I20250115 14:51:51.810622 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b68bb91c80d24bad995dc605e9702d6f" } new_table_name: "default.Table_1d1bc2260e92451c99dfebfe09374e38" modify_external_catalogs: true
I20250115 14:51:51.812747 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_662e371503c24e6cacf91cb2afe19d53" } modify_external_catalogs: true
I20250115 14:51:51.812948 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_662e371503c24e6cacf91cb2afe19d53" } modify_external_catalogs: true
I20250115 14:51:51.813288 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dd152d863f394374a58bf845d3d259e2" } modify_external_catalogs: true
I20250115 14:51:51.813434 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dd152d863f394374a58bf845d3d259e2" } modify_external_catalogs: true
I20250115 14:51:51.818547 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ec6d2615d3f348d990f55956ac678e86" } modify_external_catalogs: true
I20250115 14:51:51.818753 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ec6d2615d3f348d990f55956ac678e86" } modify_external_catalogs: true
I20250115 14:51:51.822310 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_475c872b794349c397ee73627f500616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.822825 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_475c872b794349c397ee73627f500616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.812801 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_eb8583ed6e0c445b9883e8391d53b878" } modify_external_catalogs: true
I20250115 14:51:51.823637 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_eb8583ed6e0c445b9883e8391d53b878" } modify_external_catalogs: true
I20250115 14:51:51.804239 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 718f5f0300114292bf838f5f988ab566 with delete_type TABLET_DATA_DELETED (replaced by tablet be18f1b6d8a24fabbe818811ef646c62) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.825529 1162 tablet_bootstrap.cc:492] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.825680 1162 ts_tablet_manager.cc:1397] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.006s
I20250115 14:51:51.826529 1162 raft_consensus.cc:357] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.826758 1162 raft_consensus.cc:383] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.826845 1162 raft_consensus.cc:738] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.826958 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 65f2f961e08846d09df469b74abd5da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.827092 1162 consensus_queue.cc:260] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.827541 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 101087eee11441f299d5b75e5c80b66b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.829310 1162 ts_tablet_manager.cc:1428] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:51:51.830443 1162 tablet_bootstrap.cc:492] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:51.832423 1168 tablet_bootstrap.cc:492] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.832582 1168 ts_tablet_manager.cc:1397] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.019s
I20250115 14:51:51.834429 1167 tablet_replica.cc:331] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.835129 1167 raft_consensus.cc:2238] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.835256 1162 tablet_bootstrap.cc:654] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.835359 1167 raft_consensus.cc:2267] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.833299 1168 raft_consensus.cc:357] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.836112 1167 ts_tablet_manager.cc:1905] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.836194 1168 raft_consensus.cc:383] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.836282 1168 raft_consensus.cc:738] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.836534 1168 consensus_queue.cc:260] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.838088 1168 ts_tablet_manager.cc:1428] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:51.838585 1168 tablet_bootstrap.cc:492] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.840651 1164 ts_tablet_manager.cc:1428] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.052s user 0.002s sys 0.000s
I20250115 14:51:51.841501 1168 tablet_bootstrap.cc:654] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.842592 1165 tablet_replica.cc:331] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:51:51.842630 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6a9fb932c5b4545a4609d568f0e5494 consensus state has no leader
W20250115 14:51:51.844051 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 379087c496d5401ebcbd1469ce1f68f3 consensus state has no leader
I20250115 14:51:51.844383 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_64abfdac91a54d6bb59ccfc7791a170c" } modify_external_catalogs: true
I20250115 14:51:51.844574 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_64abfdac91a54d6bb59ccfc7791a170c" } modify_external_catalogs: true
I20250115 14:51:51.845224 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 65f2f961e08846d09df469b74abd5da1 (table default.Table_0fdd97e2341d4405b18d913853a53683 [id=9fa51c9085b244b0ad70a3d49836ea39]) successfully deleted
W20250115 14:51:51.845480 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 65f2f961e08846d09df469b74abd5da1 because the tablet was not found. No further retry: Not found: Tablet not found: 65f2f961e08846d09df469b74abd5da1
W20250115 14:51:51.845808 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet be18f1b6d8a24fabbe818811ef646c62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be18f1b6d8a24fabbe818811ef646c62 already in progress: creating tablet
I20250115 14:51:51.846709 1164 tablet_bootstrap.cc:492] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:51.849918 1164 tablet_bootstrap.cc:654] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.850168 1165 raft_consensus.cc:2238] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.850637 1165 raft_consensus.cc:2267] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.852470 1165 ts_tablet_manager.cc:1905] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.856135 1167 ts_tablet_manager.cc:1918] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.850445 1166 tablet_replica.cc:331] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:51.865236 1027 tablet_service.cc:1467] Processing CreateTablet for tablet e0eba208f9be4a588cc81e4d99336eac (DEFAULT_TABLE table=default.Table_3e3a408924cc4cb0b5a408089e63bb1e [id=6d0e848982e245e1870c661f7dc35138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.865322 1166 raft_consensus.cc:2238] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.865553 1166 raft_consensus.cc:2267] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.865639 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0eba208f9be4a588cc81e4d99336eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.866405 1166 ts_tablet_manager.cc:1905] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:51.871744 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81a5e53af38c4552ae3cbd953ce57abc consensus state has no leader
I20250115 14:51:51.870373 889 tablet_service.cc:1467] Processing CreateTablet for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (DEFAULT_TABLE table=default.Table_735fcf813ef3439fa578a7fae3066b42 [id=0ef6f07cf29e46e69bf2a4449187a4e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.872187 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4161fce81834cd2ba4ffb789f6a5ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.873009 447 master_service.cc:946] ReplaceTablet: received request to replace tablet e115bdfa1d69421890122f9afdc23b29 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.874406 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 715274a681f447c594e8fcd38d6fd609 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.874786 446 master_service.cc:946] ReplaceTablet: received request to replace tablet d2529350d9264a1793c605ea9d5df9c6 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.876091 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef5e4d75580465c9c0e36135503f332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.876916 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9ddc7a7e450f465bb0113fdb6ea442ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.877439 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ddc7a7e450f465bb0113fdb6ea442ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.878188 1167 log.cc:1198] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/65f2f961e08846d09df469b74abd5da1
I20250115 14:51:51.878594 1167 ts_tablet_manager.cc:1939] T 65f2f961e08846d09df469b74abd5da1 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.879593 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 65f2f961e08846d09df469b74abd5da1 (table default.Table_0fdd97e2341d4405b18d913853a53683 [id=9fa51c9085b244b0ad70a3d49836ea39]) successfully deleted
I20250115 14:51:51.880688 1173 raft_consensus.cc:491] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:51.880859 1173 raft_consensus.cc:513] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.882786 1173 leader_election.cc:290] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:51.883061 1166 ts_tablet_manager.cc:1918] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.883582 1166 log.cc:1198] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/510ac6d6ee3e488e8b20f3e39a44ab9a
I20250115 14:51:51.883970 1166 ts_tablet_manager.cc:1939] T 510ac6d6ee3e488e8b20f3e39a44ab9a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:51.886227 1166 tablet_replica.cc:331] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:51.886449 1166 raft_consensus.cc:2238] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.886619 1166 raft_consensus.cc:2267] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.887365 1166 ts_tablet_manager.cc:1905] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.892406 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 13c00f03edc74af8b084bb236487a347
I20250115 14:51:51.893210 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 59f25f2f228143c98f6cd6ccb0cf99d7 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:51.897629 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6326b1526191464485855b657e34a2b9
I20250115 14:51:51.898835 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 6326b1526191464485855b657e34a2b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.898977 1166 ts_tablet_manager.cc:1918] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.899143 1166 log.cc:1198] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/65f2f961e08846d09df469b74abd5da1
I20250115 14:51:51.899441 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6326b1526191464485855b657e34a2b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.899504 1166 ts_tablet_manager.cc:1939] T 65f2f961e08846d09df469b74abd5da1 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:51.900440 1166 tablet_replica.cc:331] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:51.900626 1166 raft_consensus.cc:2238] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.900792 1166 raft_consensus.cc:2267] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.901974 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c00f03edc74af8b084bb236487a347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.902570 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c00f03edc74af8b084bb236487a347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.906960 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4e0b412e80014674bc22990db5953cc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.907647 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e0b412e80014674bc22990db5953cc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.910717 754 tablet_service.cc:1467] Processing CreateTablet for tablet e0eba208f9be4a588cc81e4d99336eac (DEFAULT_TABLE table=default.Table_3e3a408924cc4cb0b5a408089e63bb1e [id=6d0e848982e245e1870c661f7dc35138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.911154 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0eba208f9be4a588cc81e4d99336eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.916070 1026 tablet_service.cc:1467] Processing CreateTablet for tablet ad67354f35ce4564b88dbdffcabf3564 (DEFAULT_TABLE table=default.Table_bbf01a6c24b9449b8da3ee9428b7bd98 [id=23302b50545244ddb17c074740aacccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.916623 889 tablet_service.cc:1467] Processing CreateTablet for tablet ad67354f35ce4564b88dbdffcabf3564 (DEFAULT_TABLE table=default.Table_bbf01a6c24b9449b8da3ee9428b7bd98 [id=23302b50545244ddb17c074740aacccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.916708 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad67354f35ce4564b88dbdffcabf3564. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.917032 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad67354f35ce4564b88dbdffcabf3564. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.922382 891 tablet_service.cc:1467] Processing CreateTablet for tablet f960c88df14242c5ada1ecf9d8e68196 (DEFAULT_TABLE table=default.Table_de5e692c0fa544a5975b1cc474c6a6d5 [id=24d4e203bce64fcdbd5b6fa30a99e6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.922761 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f960c88df14242c5ada1ecf9d8e68196. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.923501 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c00f03edc74af8b084bb236487a347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.923842 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 6326b1526191464485855b657e34a2b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.924990 754 tablet_service.cc:1467] Processing CreateTablet for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (DEFAULT_TABLE table=default.Table_735fcf813ef3439fa578a7fae3066b42 [id=0ef6f07cf29e46e69bf2a4449187a4e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.925336 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4161fce81834cd2ba4ffb789f6a5ba3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:51.928786 710 leader_election.cc:343] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 101087eee11441f299d5b75e5c80b66b
W20250115 14:51:51.929236 710 leader_election.cc:343] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 101087eee11441f299d5b75e5c80b66b
I20250115 14:51:51.929363 710 leader_election.cc:304] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:51.929661 710 raft_consensus.cc:2681] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:51:51.930436 1166 ts_tablet_manager.cc:1905] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.932709 884 tablet_service.cc:1467] Processing CreateTablet for tablet e0eba208f9be4a588cc81e4d99336eac (DEFAULT_TABLE table=default.Table_3e3a408924cc4cb0b5a408089e63bb1e [id=6d0e848982e245e1870c661f7dc35138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.933161 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0eba208f9be4a588cc81e4d99336eac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.936089 1166 ts_tablet_manager.cc:1918] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.936204 1166 log.cc:1198] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/101087eee11441f299d5b75e5c80b66b
I20250115 14:51:51.936868 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 510ac6d6ee3e488e8b20f3e39a44ab9a (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]) successfully deleted
I20250115 14:51:51.937125 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 65f2f961e08846d09df469b74abd5da1 (table default.Table_0fdd97e2341d4405b18d913853a53683 [id=9fa51c9085b244b0ad70a3d49836ea39]) successfully deleted
I20250115 14:51:51.938866 1167 tablet_replica.cc:331] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.939065 1167 raft_consensus.cc:2238] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.939224 1167 raft_consensus.cc:2267] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.939970 1167 ts_tablet_manager.cc:1905] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.940824 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 85ab1ccf420d41bd98cfda29caf5a619
I20250115 14:51:51.940927 1165 ts_tablet_manager.cc:1918] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.941054 1165 log.cc:1198] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f1328f038ad3423a8f0edcb13e0f555a
I20250115 14:51:51.941411 1165 ts_tablet_manager.cc:1939] T f1328f038ad3423a8f0edcb13e0f555a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:51.941562 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 8d43cd66b18f4649894a629637037822
I20250115 14:51:51.941665 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e115bdfa1d69421890122f9afdc23b29
I20250115 14:51:51.941967 455 catalog_manager.cc:6596] ReplaceTablet: tablet 8d43cd66b18f4649894a629637037822 deleted and replaced by tablet bb792ca1b3f54def92f504ec8682d2c7
I20250115 14:51:51.942067 447 catalog_manager.cc:6596] ReplaceTablet: tablet e115bdfa1d69421890122f9afdc23b29 deleted and replaced by tablet f312f1e3a33a404ca7284c713859f156
I20250115 14:51:51.942224 1174 raft_consensus.cc:491] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:51.942339 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c322301320ba425a94d7321c08f7f9e6
I20250115 14:51:51.942360 1174 raft_consensus.cc:513] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.943014 1174 leader_election.cc:290] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:51.943570 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a5e53af38c4552ae3cbd953ce57abc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:51.943655 1027 tablet_service.cc:1467] Processing CreateTablet for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (DEFAULT_TABLE table=default.Table_735fcf813ef3439fa578a7fae3066b42 [id=0ef6f07cf29e46e69bf2a4449187a4e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.943830 1049 raft_consensus.cc:2463] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:51.943941 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet e115bdfa1d69421890122f9afdc23b29 with delete_type TABLET_DATA_DELETED (replaced by tablet f312f1e3a33a404ca7284c713859f156) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.944146 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4161fce81834cd2ba4ffb789f6a5ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.944181 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet c322301320ba425a94d7321c08f7f9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.944247 847 leader_election.cc:304] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:51.943825 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d43cd66b18f4649894a629637037822 with delete_type TABLET_DATA_DELETED (replaced by tablet bb792ca1b3f54def92f504ec8682d2c7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.944532 1174 raft_consensus.cc:2798] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:51.944620 1174 raft_consensus.cc:491] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:51.944733 1174 raft_consensus.cc:3054] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:51.945206 888 tablet_service.cc:1514] Processing DeleteTablet for tablet e115bdfa1d69421890122f9afdc23b29 with delete_type TABLET_DATA_DELETED (replaced by tablet f312f1e3a33a404ca7284c713859f156) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.945628 888 tablet_service.cc:1514] Processing DeleteTablet for tablet c322301320ba425a94d7321c08f7f9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.946098 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d2529350d9264a1793c605ea9d5df9c6
I20250115 14:51:51.946480 446 catalog_manager.cc:6596] ReplaceTablet: tablet d2529350d9264a1793c605ea9d5df9c6 deleted and replaced by tablet fad6f9f7f853418e85353edfd38cc80e
I20250115 14:51:51.947088 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet d2529350d9264a1793c605ea9d5df9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fad6f9f7f853418e85353edfd38cc80e) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.947204 1174 raft_consensus.cc:513] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.947535 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d2529350d9264a1793c605ea9d5df9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fad6f9f7f853418e85353edfd38cc80e) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.948200 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a5e53af38c4552ae3cbd953ce57abc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:51.948405 1049 raft_consensus.cc:3054] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:51.948650 1021 tablet_service.cc:1467] Processing CreateTablet for tablet f960c88df14242c5ada1ecf9d8e68196 (DEFAULT_TABLE table=default.Table_de5e692c0fa544a5975b1cc474c6a6d5 [id=24d4e203bce64fcdbd5b6fa30a99e6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.949012 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f960c88df14242c5ada1ecf9d8e68196. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.951339 1049 raft_consensus.cc:2463] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:51.951709 847 leader_election.cc:304] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:51.951990 1209 raft_consensus.cc:2798] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:51.952014 1174 leader_election.cc:290] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:51.952144 1209 raft_consensus.cc:695] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:51.952467 1209 consensus_queue.cc:237] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.955595 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d43cd66b18f4649894a629637037822 with delete_type TABLET_DATA_DELETED (replaced by tablet bb792ca1b3f54def92f504ec8682d2c7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.956519 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 379087c496d5401ebcbd1469ce1f68f3 (table default.Table_1d1bc2260e92451c99dfebfe09374e38 [id=959c2861880b40dcbb3780f3dff3b0f9]). Expected version 1 got 0
I20250115 14:51:51.956745 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ab1ccf420d41bd98cfda29caf5a619 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.957119 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet edd50ffca1ee49d0bbe96187480ea26b
I20250115 14:51:51.958364 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a5e53af38c4552ae3cbd953ce57abc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:51.958753 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet edd50ffca1ee49d0bbe96187480ea26b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.958793 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a5e53af38c4552ae3cbd953ce57abc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:51.959088 891 tablet_service.cc:1514] Processing DeleteTablet for tablet edd50ffca1ee49d0bbe96187480ea26b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:51.959302 753 tablet_service.cc:1467] Processing CreateTablet for tablet f960c88df14242c5ada1ecf9d8e68196 (DEFAULT_TABLE table=default.Table_de5e692c0fa544a5975b1cc474c6a6d5 [id=24d4e203bce64fcdbd5b6fa30a99e6b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.959626 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ab1ccf420d41bd98cfda29caf5a619 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:51.959719 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f960c88df14242c5ada1ecf9d8e68196. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.961086 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_86166d8e2b8e40f59fc59eb5d8076fbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.961753 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86166d8e2b8e40f59fc59eb5d8076fbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.962116 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_075e350a2fa14e2eb822f40144f72eac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:51.963725 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_075e350a2fa14e2eb822f40144f72eac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:51.964543 1225 raft_consensus.cc:491] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:51.964710 1225 raft_consensus.cc:513] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.965313 1225 leader_election.cc:290] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:51.965881 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ab1ccf420d41bd98cfda29caf5a619" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:51.966041 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ab1ccf420d41bd98cfda29caf5a619" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:51.966233 912 raft_consensus.cc:2463] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
W20250115 14:51:51.966373 984 leader_election.cc:343] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.966619 983 leader_election.cc:304] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:51.967074 1225 raft_consensus.cc:2798] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:51.967170 1225 raft_consensus.cc:491] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:51.967270 1225 raft_consensus.cc:3054] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:51.969959 1225 raft_consensus.cc:513] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.970820 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ab1ccf420d41bd98cfda29caf5a619" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:51.971056 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85ab1ccf420d41bd98cfda29caf5a619" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:51.971226 912 raft_consensus.cc:3054] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:51.971290 984 leader_election.cc:343] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.973654 912 raft_consensus.cc:2463] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:51.974197 1169 raft_consensus.cc:491] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:51.974227 983 leader_election.cc:304] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:51.974346 1169 raft_consensus.cc:513] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.974923 1169 leader_election.cc:290] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127)
I20250115 14:51:51.975075 1169 raft_consensus.cc:2798] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:51.975214 1169 raft_consensus.cc:695] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:51.975306 1225 leader_election.cc:290] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:51.975486 1169 consensus_queue.cc:237] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.977159 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef5e4d75580465c9c0e36135503f332" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:51.977727 984 leader_election.cc:343] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.977810 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ab1ccf420d41bd98cfda29caf5a619 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:51.978416 848 leader_election.cc:343] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:51.978576 983 leader_election.cc:343] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 3ef5e4d75580465c9c0e36135503f332
I20250115 14:51:51.978682 983 leader_election.cc:304] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:51.978734 848 leader_election.cc:343] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:51.979020 1226 raft_consensus.cc:2743] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:51.979233 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d43cd66b18f4649894a629637037822 with delete_type TABLET_DATA_DELETED (replaced by tablet bb792ca1b3f54def92f504ec8682d2c7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.979584 753 tablet_service.cc:1514] Processing DeleteTablet for tablet e115bdfa1d69421890122f9afdc23b29 with delete_type TABLET_DATA_DELETED (replaced by tablet f312f1e3a33a404ca7284c713859f156) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.979933 753 tablet_service.cc:1514] Processing DeleteTablet for tablet c322301320ba425a94d7321c08f7f9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.980418 753 tablet_service.cc:1514] Processing DeleteTablet for tablet d2529350d9264a1793c605ea9d5df9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fad6f9f7f853418e85353edfd38cc80e) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.980749 753 tablet_service.cc:1514] Processing DeleteTablet for tablet edd50ffca1ee49d0bbe96187480ea26b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:51.982915 1162 tablet_bootstrap.cc:492] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:51.983111 1162 ts_tablet_manager.cc:1397] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.153s user 0.032s sys 0.046s
I20250115 14:51:51.983984 1162 raft_consensus.cc:357] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.984218 1162 raft_consensus.cc:383] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.984304 1162 raft_consensus.cc:738] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.984558 1164 tablet_bootstrap.cc:492] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:51.984750 1164 ts_tablet_manager.cc:1397] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.138s user 0.003s sys 0.002s
I20250115 14:51:51.985268 1168 tablet_bootstrap.cc:492] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.985432 1168 ts_tablet_manager.cc:1397] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.147s user 0.023s sys 0.035s
I20250115 14:51:51.985529 1164 raft_consensus.cc:357] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.985838 1164 raft_consensus.cc:383] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.985997 1164 raft_consensus.cc:738] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.986215 1164 consensus_queue.cc:260] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.986335 1168 raft_consensus.cc:357] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.986555 1168 raft_consensus.cc:383] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.986646 1168 raft_consensus.cc:738] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.986704 1167 ts_tablet_manager.cc:1918] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.986833 1167 log.cc:1198] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/59f25f2f228143c98f6cd6ccb0cf99d7
I20250115 14:51:51.986878 1168 consensus_queue.cc:260] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.987186 1167 ts_tablet_manager.cc:1939] T 59f25f2f228143c98f6cd6ccb0cf99d7 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:51.987704 1168 ts_tablet_manager.cc:1428] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:51.987763 1164 ts_tablet_manager.cc:1428] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250115 14:51:51.987959 1168 tablet_bootstrap.cc:492] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:51.988025 1167 tablet_replica.cc:331] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:51.988034 1164 tablet_bootstrap.cc:492] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:51.988178 1167 raft_consensus.cc:2238] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.988358 1167 raft_consensus.cc:2267] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.988817 1162 consensus_queue.cc:260] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.990185 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f1328f038ad3423a8f0edcb13e0f555a (table default.Table_1b3a658f6eda417cb956eba9cdc5a2af [id=d7ba4cda40964e1d839efe9e0f1615c0]) successfully deleted
I20250115 14:51:51.990568 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 59f25f2f228143c98f6cd6ccb0cf99d7 (table default.Table_7de6e49996a0432d8f8bfa56c51d0782 [id=ff5b7f4d675f496eaf7dd60b0cca7435]) successfully deleted
I20250115 14:51:51.990873 1165 tablet_replica.cc:331] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:51.991065 1165 raft_consensus.cc:2238] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:51.991216 1162 ts_tablet_manager.cc:1428] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20250115 14:51:51.991233 1165 raft_consensus.cc:2267] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:51.991907 1165 ts_tablet_manager.cc:1905] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:51.991969 1168 tablet_bootstrap.cc:654] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.992015 1164 tablet_bootstrap.cc:654] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:51.992818 754 tablet_service.cc:1467] Processing CreateTablet for tablet ad67354f35ce4564b88dbdffcabf3564 (DEFAULT_TABLE table=default.Table_bbf01a6c24b9449b8da3ee9428b7bd98 [id=23302b50545244ddb17c074740aacccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:51.993196 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad67354f35ce4564b88dbdffcabf3564. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:51.995764 1168 tablet_bootstrap.cc:492] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:51.995909 1168 ts_tablet_manager.cc:1397] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:51:51.996565 1168 raft_consensus.cc:357] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.996801 1168 raft_consensus.cc:383] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:51.996872 1168 raft_consensus.cc:738] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:51.998544 1165 ts_tablet_manager.cc:1918] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:51.998672 1165 log.cc:1198] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/59f25f2f228143c98f6cd6ccb0cf99d7
I20250115 14:51:51.998719 1168 consensus_queue.cc:260] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:51.999022 1165 ts_tablet_manager.cc:1939] T 59f25f2f228143c98f6cd6ccb0cf99d7 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:51.999217 1166 ts_tablet_manager.cc:1939] T 101087eee11441f299d5b75e5c80b66b P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.000677 1168 ts_tablet_manager.cc:1428] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:51:52.001184 1162 tablet_bootstrap.cc:492] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.001300 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ca27ab47d077491489383629eaaf1437"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.001928 1168 tablet_bootstrap.cc:492] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.003265 1162 tablet_bootstrap.cc:654] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.003357 1168 tablet_bootstrap.cc:654] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.005168 1167 ts_tablet_manager.cc:1905] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:52.008383 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca27ab47d077491489383629eaaf1437 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.010883 1162 tablet_bootstrap.cc:492] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.010977 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 59f25f2f228143c98f6cd6ccb0cf99d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.011026 1162 ts_tablet_manager.cc:1397] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20250115 14:51:52.011195 1166 tablet_replica.cc:331] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.011611 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 101087eee11441f299d5b75e5c80b66b (table default.Table_f04c4734007f45f2b1b6dbe3cfeec857 [id=05c9f91a89fb43c8918c6cfe74360e43]) successfully deleted
I20250115 14:51:52.011382 1166 raft_consensus.cc:2238] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.011762 1162 raft_consensus.cc:357] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.011994 1162 raft_consensus.cc:383] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.012104 1162 raft_consensus.cc:738] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.012384 1162 consensus_queue.cc:260] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.012923 1162 ts_tablet_manager.cc:1428] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:52.013612 1165 tablet_replica.cc:331] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.013792 1165 raft_consensus.cc:2238] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.013942 1165 raft_consensus.cc:2267] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.013950 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 59f25f2f228143c98f6cd6ccb0cf99d7 (table default.Table_7de6e49996a0432d8f8bfa56c51d0782 [id=ff5b7f4d675f496eaf7dd60b0cca7435]) successfully deleted
W20250115 14:51:52.014222 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2b4259dfb8ee4b618c0bacc910addd0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b4259dfb8ee4b618c0bacc910addd0a already in progress: creating tablet
I20250115 14:51:52.014452 1165 ts_tablet_manager.cc:1905] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.014489 1166 raft_consensus.cc:2267] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.015184 1166 ts_tablet_manager.cc:1905] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.018437 1162 tablet_bootstrap.cc:492] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:52.018836 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 13c00f03edc74af8b084bb236487a347 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13c00f03edc74af8b084bb236487a347 already in progress: creating tablet
W20250115 14:51:52.019165 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6326b1526191464485855b657e34a2b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6326b1526191464485855b657e34a2b9 already in progress: creating tablet
W20250115 14:51:52.019879 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 85ab1ccf420d41bd98cfda29caf5a619 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85ab1ccf420d41bd98cfda29caf5a619 already in progress: creating tablet
W20250115 14:51:52.020156 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 8d43cd66b18f4649894a629637037822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d43cd66b18f4649894a629637037822 already in progress: creating tablet
W20250115 14:51:52.020417 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e115bdfa1d69421890122f9afdc23b29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e115bdfa1d69421890122f9afdc23b29 already in progress: creating tablet
I20250115 14:51:52.022058 1162 tablet_bootstrap.cc:654] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:52.022292 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c322301320ba425a94d7321c08f7f9e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c322301320ba425a94d7321c08f7f9e6 already in progress: creating tablet
W20250115 14:51:52.022545 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d2529350d9264a1793c605ea9d5df9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2529350d9264a1793c605ea9d5df9c6 already in progress: creating tablet
W20250115 14:51:52.022796 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet edd50ffca1ee49d0bbe96187480ea26b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edd50ffca1ee49d0bbe96187480ea26b already in progress: creating tablet
I20250115 14:51:52.024382 1167 ts_tablet_manager.cc:1918] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.024464 1164 tablet_bootstrap.cc:492] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.024507 1167 log.cc:1198] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/1d0d566e954045fc85c4fbfe8052b050
I20250115 14:51:52.024624 1164 ts_tablet_manager.cc:1397] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.000s
I20250115 14:51:52.024605 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_075e350a2fa14e2eb822f40144f72eac" } new_table_name: "default.Table_3afc89f290d048999c9654eeed8bb849" modify_external_catalogs: true
I20250115 14:51:52.025383 1164 raft_consensus.cc:357] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.025584 1164 raft_consensus.cc:383] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.025651 1164 raft_consensus.cc:738] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.025838 1167 ts_tablet_manager.cc:1939] T 1d0d566e954045fc85c4fbfe8052b050 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.025933 1164 consensus_queue.cc:260] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.026732 1168 tablet_bootstrap.cc:492] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.026638 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b4f282a7f918430ab5956c52b4e7f7fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.026870 1168 ts_tablet_manager.cc:1397] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
W20250115 14:51:52.027132 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4f282a7f918430ab5956c52b4e7f7fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.027518 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 1d0d566e954045fc85c4fbfe8052b050 (table default.Table_fb13e7bf7ead4609a5d43f200ab36cad [id=a415575b540a4472bc26682438479572]) successfully deleted
I20250115 14:51:52.027554 1168 raft_consensus.cc:357] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.028071 1165 ts_tablet_manager.cc:1918] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.028220 1165 log.cc:1198] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/649504c31def4f3f9c30879033b58fbe
I20250115 14:51:52.028270 1168 raft_consensus.cc:383] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.028393 1168 raft_consensus.cc:738] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.028569 1165 ts_tablet_manager.cc:1939] T 649504c31def4f3f9c30879033b58fbe P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.028616 1168 consensus_queue.cc:260] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.030102 1164 ts_tablet_manager.cc:1428] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:51:52.031132 1168 ts_tablet_manager.cc:1428] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250115 14:51:52.031369 1167 tablet_replica.cc:331] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.031554 1167 raft_consensus.cc:2238] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.031699 1167 raft_consensus.cc:2267] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.032665 1168 tablet_bootstrap.cc:492] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.033221 1165 tablet_replica.cc:331] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.033519 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 649504c31def4f3f9c30879033b58fbe (table default.Table_c1f78195e62c4c148f6b298e2cbc055e [id=f6f31e321abb451c877540a31b2fb976]) successfully deleted
I20250115 14:51:52.033877 1165 raft_consensus.cc:2238] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.033891 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3921752ce1634998826bf8a3513582d9" } new_table_name: "default.Table_2e17c919cc6f4992b7d7b764ba964949" modify_external_catalogs: true
I20250115 14:51:52.034034 1165 raft_consensus.cc:2267] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.034422 1168 tablet_bootstrap.cc:654] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.034513 1165 ts_tablet_manager.cc:1905] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.035084 1164 tablet_bootstrap.cc:492] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.032665 1167 ts_tablet_manager.cc:1905] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.036329 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c87fa8eb006845f5bf7b37b90f4c14c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.036795 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c87fa8eb006845f5bf7b37b90f4c14c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.039091 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4b96cde69a5b411f97fe0f70ae0264c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.039288 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dfdad82a801948d08483f75e70f61100"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.039564 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b96cde69a5b411f97fe0f70ae0264c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:52.039778 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfdad82a801948d08483f75e70f61100 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.040796 1166 ts_tablet_manager.cc:1918] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.040938 1166 log.cc:1198] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/59f25f2f228143c98f6cd6ccb0cf99d7
I20250115 14:51:52.041313 1166 ts_tablet_manager.cc:1939] T 59f25f2f228143c98f6cd6ccb0cf99d7 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.045974 1173 raft_consensus.cc:491] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.046139 1173 raft_consensus.cc:513] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.046222 1167 ts_tablet_manager.cc:1918] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.046370 1167 log.cc:1198] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/718f5f0300114292bf838f5f988ab566
I20250115 14:51:52.046800 1173 leader_election.cc:290] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.047322 1167 ts_tablet_manager.cc:1939] T 718f5f0300114292bf838f5f988ab566 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
W20250115 14:51:52.047968 710 leader_election.cc:343] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 1d0d566e954045fc85c4fbfe8052b050
I20250115 14:51:52.048162 1226 raft_consensus.cc:491] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.048578 1164 tablet_bootstrap.cc:654] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.048341 1226 raft_consensus.cc:513] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.049715 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 718f5f0300114292bf838f5f988ab566 (table default.Table_40664b92268d4ba39f3f5852835f97d2 [id=da39f1615db74469824c3bf8eb74d260]) successfully deleted
I20250115 14:51:52.050051 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379087c496d5401ebcbd1469ce1f68f3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:52.050328 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0d566e954045fc85c4fbfe8052b050" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:52.050609 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 59f25f2f228143c98f6cd6ccb0cf99d7 (table default.Table_7de6e49996a0432d8f8bfa56c51d0782 [id=ff5b7f4d675f496eaf7dd60b0cca7435]) successfully deleted
I20250115 14:51:52.052239 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379087c496d5401ebcbd1469ce1f68f3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:52.052490 912 raft_consensus.cc:2463] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
W20250115 14:51:52.052482 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6326b1526191464485855b657e34a2b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6326b1526191464485855b657e34a2b9 already in progress: creating tablet
I20250115 14:51:52.053299 1226 leader_election.cc:290] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:52.053360 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 13c00f03edc74af8b084bb236487a347 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13c00f03edc74af8b084bb236487a347 already in progress: creating tablet
W20250115 14:51:52.053736 984 leader_election.cc:343] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.054229 983 leader_election.cc:304] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:52.054311 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e115bdfa1d69421890122f9afdc23b29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e115bdfa1d69421890122f9afdc23b29 already in progress: creating tablet
W20250115 14:51:52.054982 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c322301320ba425a94d7321c08f7f9e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c322301320ba425a94d7321c08f7f9e6 already in progress: creating tablet
I20250115 14:51:52.055238 1226 raft_consensus.cc:2798] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:52.055341 1226 raft_consensus.cc:491] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:52.055418 1226 raft_consensus.cc:3054] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:52.055579 710 leader_election.cc:304] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:52.055830 1168 tablet_bootstrap.cc:492] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.055881 1173 raft_consensus.cc:2743] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:52.055976 1168 ts_tablet_manager.cc:1397] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.004s
W20250115 14:51:52.056360 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fd72a0d586d4a2c817f6a39daa07063 has no consensus state
I20250115 14:51:52.056737 1168 raft_consensus.cc:357] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.056934 1168 raft_consensus.cc:383] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.057018 1168 raft_consensus.cc:738] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
W20250115 14:51:52.057116 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70e2495496b746abb568e78377c83d1e consensus state has no leader
I20250115 14:51:52.057271 1168 consensus_queue.cc:260] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.058151 1209 raft_consensus.cc:491] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.058285 1209 raft_consensus.cc:513] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.058737 1226 raft_consensus.cc:513] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.058838 1209 leader_election.cc:290] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.059592 1168 ts_tablet_manager.cc:1428] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:51:52.059644 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379087c496d5401ebcbd1469ce1f68f3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:52.060200 984 leader_election.cc:343] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.060364 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd50ffca1ee49d0bbe96187480ea26b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:52.060896 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd50ffca1ee49d0bbe96187480ea26b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:52.060506 453 catalog_manager.cc:5526] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:52.061072 1049 raft_consensus.cc:2463] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:52.061292 1168 tablet_bootstrap.cc:492] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.061336 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "379087c496d5401ebcbd1469ce1f68f3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:52.061405 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 81a5e53af38c4552ae3cbd953ce57abc (table default.Table_47396588856f48a6ae9cb4d6469a39d1 [id=9db6edcf4cfb491580916c914e61de0c]). Expected version 1 got 0
I20250115 14:51:52.061558 912 raft_consensus.cc:3054] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:52.062319 847 leader_election.cc:304] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:51:52.062517 848 leader_election.cc:343] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.062656 1167 tablet_replica.cc:331] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.062849 1167 raft_consensus.cc:2238] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.062877 1209 raft_consensus.cc:2798] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:52.062976 1209 raft_consensus.cc:491] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:52.063043 1167 raft_consensus.cc:2267] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:52.063048 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d2529350d9264a1793c605ea9d5df9c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2529350d9264a1793c605ea9d5df9c6 already in progress: creating tablet
I20250115 14:51:52.063091 1209 raft_consensus.cc:3054] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:52.063266 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 8d43cd66b18f4649894a629637037822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d43cd66b18f4649894a629637037822 already in progress: creating tablet
I20250115 14:51:52.063575 1167 ts_tablet_manager.cc:1905] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.064263 912 raft_consensus.cc:2463] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:52.064854 983 leader_election.cc:304] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:52.065142 1169 raft_consensus.cc:2798] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:52.065255 1169 raft_consensus.cc:695] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:52.065511 1169 consensus_queue.cc:237] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.065824 1209 raft_consensus.cc:513] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.066473 1026 tablet_service.cc:1467] Processing CreateTablet for tablet d77ff794a83348c29391786666987db1 (DEFAULT_TABLE table=default.Table_9ddc7a7e450f465bb0113fdb6ea442ed [id=6c9d98b8d6de494a9c6211022d1fc946]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.066800 884 tablet_service.cc:1467] Processing CreateTablet for tablet d77ff794a83348c29391786666987db1 (DEFAULT_TABLE table=default.Table_9ddc7a7e450f465bb0113fdb6ea442ed [id=6c9d98b8d6de494a9c6211022d1fc946]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.066869 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d77ff794a83348c29391786666987db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.066967 1168 tablet_bootstrap.cc:654] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.067152 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d77ff794a83348c29391786666987db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.068027 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd50ffca1ee49d0bbe96187480ea26b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:52.068212 1049 raft_consensus.cc:3054] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:52.070855 1049 raft_consensus.cc:2463] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:52.071509 847 leader_election.cc:304] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:52.071801 1207 raft_consensus.cc:2798] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:52.071926 1207 raft_consensus.cc:695] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:52.072188 1207 consensus_queue.cc:237] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.073107 1226 leader_election.cc:290] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:52.073511 1209 leader_election.cc:290] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.074293 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd50ffca1ee49d0bbe96187480ea26b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:52.075222 848 leader_election.cc:343] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.075438 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c56fe5b5b59b4017be3cb0ac877479b4 (DEFAULT_TABLE table=default.Table_22fff0dd323f40378324128fc8bdddd8 [id=3e278eb470174b78a8cdaab8d9ab2464]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.076097 891 tablet_service.cc:1467] Processing CreateTablet for tablet c56fe5b5b59b4017be3cb0ac877479b4 (DEFAULT_TABLE table=default.Table_22fff0dd323f40378324128fc8bdddd8 [id=3e278eb470174b78a8cdaab8d9ab2464]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.076465 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c56fe5b5b59b4017be3cb0ac877479b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.081192 1162 tablet_bootstrap.cc:492] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.081318 1164 tablet_bootstrap.cc:492] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.081357 1162 ts_tablet_manager.cc:1397] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.005s
I20250115 14:51:52.081465 1164 ts_tablet_manager.cc:1397] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.000s
I20250115 14:51:52.082098 1162 raft_consensus.cc:357] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.082190 1164 raft_consensus.cc:357] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.082330 1162 raft_consensus.cc:383] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.082369 1164 raft_consensus.cc:383] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.082407 1162 raft_consensus.cc:738] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.082456 1164 raft_consensus.cc:738] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.082671 1164 consensus_queue.cc:260] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.075786 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c56fe5b5b59b4017be3cb0ac877479b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.084195 1164 ts_tablet_manager.cc:1428] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:52.084626 1164 tablet_bootstrap.cc:492] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.088193 1164 tablet_bootstrap.cc:654] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.088421 1162 consensus_queue.cc:260] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.090533 1167 ts_tablet_manager.cc:1918] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.090672 1167 log.cc:1198] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/85ab1ccf420d41bd98cfda29caf5a619
I20250115 14:51:52.091063 1167 ts_tablet_manager.cc:1939] T 85ab1ccf420d41bd98cfda29caf5a619 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.090678 891 tablet_service.cc:1467] Processing CreateTablet for tablet 24751e606ad94675b650d67a3738481e (DEFAULT_TABLE table=default.Table_49eb379e719f4677b13269d15ef7efad [id=af78a0b6494d4fd68224992dec097051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.091768 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24751e606ad94675b650d67a3738481e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.092054 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 85ab1ccf420d41bd98cfda29caf5a619 (table default.Table_eb8583ed6e0c445b9883e8391d53b878 [id=af4d0a3d017c4e2cbc04fa416fa68e5a]) successfully deleted
I20250115 14:51:52.095763 1162 ts_tablet_manager.cc:1428] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250115 14:51:52.096149 754 tablet_service.cc:1467] Processing CreateTablet for tablet d77ff794a83348c29391786666987db1 (DEFAULT_TABLE table=default.Table_9ddc7a7e450f465bb0113fdb6ea442ed [id=6c9d98b8d6de494a9c6211022d1fc946]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.096452 1164 tablet_bootstrap.cc:492] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.096576 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d77ff794a83348c29391786666987db1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.096614 1164 ts_tablet_manager.cc:1397] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250115 14:51:52.096905 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 24751e606ad94675b650d67a3738481e (DEFAULT_TABLE table=default.Table_49eb379e719f4677b13269d15ef7efad [id=af78a0b6494d4fd68224992dec097051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.097286 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24751e606ad94675b650d67a3738481e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.097803 1167 tablet_replica.cc:331] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.098537 1164 raft_consensus.cc:357] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.098714 1162 tablet_bootstrap.cc:492] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.098755 1164 raft_consensus.cc:383] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.098826 1164 raft_consensus.cc:738] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.102556 753 tablet_service.cc:1467] Processing CreateTablet for tablet c56fe5b5b59b4017be3cb0ac877479b4 (DEFAULT_TABLE table=default.Table_22fff0dd323f40378324128fc8bdddd8 [id=3e278eb470174b78a8cdaab8d9ab2464]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.102941 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c56fe5b5b59b4017be3cb0ac877479b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.103693 1168 tablet_bootstrap.cc:492] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.103822 1168 ts_tablet_manager.cc:1397] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.005s
I20250115 14:51:52.104575 1168 raft_consensus.cc:357] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.104759 1168 raft_consensus.cc:383] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.104838 1168 raft_consensus.cc:738] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.105067 1168 consensus_queue.cc:260] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.105516 1168 ts_tablet_manager.cc:1428] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:52.105775 1168 tablet_bootstrap.cc:492] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.116909 1162 tablet_bootstrap.cc:654] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.118440 1168 tablet_bootstrap.cc:654] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.119540 1164 consensus_queue.cc:260] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.121920 1165 ts_tablet_manager.cc:1918] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.122144 1165 log.cc:1198] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/0e20ad209d7044b09935691f002b9998
I20250115 14:51:52.122561 1165 ts_tablet_manager.cc:1939] T 0e20ad209d7044b09935691f002b9998 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.123586 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 0e20ad209d7044b09935691f002b9998 (table default.Table_5d2e74e77f87456396aed66d4d21f840 [id=1fc8e792fbbe4faf85699a5a99b99908]) successfully deleted
I20250115 14:51:52.124060 1165 tablet_replica.cc:331] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.124341 1165 raft_consensus.cc:2238] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:52.124545 1164 ts_tablet_manager.cc:1428] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20250115 14:51:52.124869 1165 pending_rounds.cc:62] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:52.125123 1165 pending_rounds.cc:69] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558307299033088 op_type: NO_OP noop_request { }
I20250115 14:51:52.125358 1165 raft_consensus.cc:2883] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:52.125510 1165 raft_consensus.cc:2267] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.126585 1164 tablet_bootstrap.cc:492] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.127174 1165 ts_tablet_manager.cc:1905] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.128057 1164 tablet_bootstrap.cc:654] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.129617 1168 tablet_bootstrap.cc:492] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.129801 1168 ts_tablet_manager.cc:1397] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20250115 14:51:52.130656 1168 raft_consensus.cc:357] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.130889 1168 raft_consensus.cc:383] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.130980 1168 raft_consensus.cc:738] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.131299 1168 consensus_queue.cc:260] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.132038 1168 ts_tablet_manager.cc:1428] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:52.132341 1168 tablet_bootstrap.cc:492] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.133176 884 tablet_service.cc:1467] Processing CreateTablet for tablet 69aed449a776493b8bdc7a31b47b62e8 (DEFAULT_TABLE table=default.Table_68b340d318b14c078d634b22c471d979 [id=e2d228ef5e814fd9bf50e2ed7e89be4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.133565 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69aed449a776493b8bdc7a31b47b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.134204 1162 tablet_bootstrap.cc:492] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.134332 1162 ts_tablet_manager.cc:1397] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20250115 14:51:52.134610 1168 tablet_bootstrap.cc:654] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.135030 1162 raft_consensus.cc:357] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.135216 1162 raft_consensus.cc:383] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.135290 1162 raft_consensus.cc:738] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.135494 1162 consensus_queue.cc:260] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.136037 1162 ts_tablet_manager.cc:1428] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:52.136369 1162 tablet_bootstrap.cc:492] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.138059 1162 tablet_bootstrap.cc:654] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.138979 1165 ts_tablet_manager.cc:1918] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:52.139101 1165 log.cc:1198] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/1d0d566e954045fc85c4fbfe8052b050
I20250115 14:51:52.139426 1165 ts_tablet_manager.cc:1939] T 1d0d566e954045fc85c4fbfe8052b050 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.139310 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_735fcf813ef3439fa578a7fae3066b42" } new_table_name: "default.Table_881f3ef58e824b4cb4e1fb3ec4846cc7" modify_external_catalogs: true
I20250115 14:51:52.140384 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 1d0d566e954045fc85c4fbfe8052b050 (table default.Table_fb13e7bf7ead4609a5d43f200ab36cad [id=a415575b540a4472bc26682438479572]) successfully deleted
I20250115 14:51:52.143492 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3b4117171441a987ac043931fc6897 (DEFAULT_TABLE table=default.Table_8d37d2aba1094de3a9bc2eabee94643d [id=207a93a004da4fbc91db802c6939059a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.143496 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3b4117171441a987ac043931fc6897 (DEFAULT_TABLE table=default.Table_8d37d2aba1094de3a9bc2eabee94643d [id=207a93a004da4fbc91db802c6939059a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.143550 888 tablet_service.cc:1467] Processing CreateTablet for tablet bb792ca1b3f54def92f504ec8682d2c7 (DEFAULT_TABLE table=default.Table_733144094e85412998e706e7a0f93dbf [id=e1e34c5d139f4810a8f45a223e459e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.143862 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d3b4117171441a987ac043931fc6897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.143890 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d3b4117171441a987ac043931fc6897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.143919 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 69aed449a776493b8bdc7a31b47b62e8 (DEFAULT_TABLE table=default.Table_68b340d318b14c078d634b22c471d979 [id=e2d228ef5e814fd9bf50e2ed7e89be4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.144214 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69aed449a776493b8bdc7a31b47b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.144515 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb792ca1b3f54def92f504ec8682d2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.146354 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e1192ebeb25e40cbaf8670c6cf0d5f76" } new_table_name: "default.Table_2752e5f3a4ca4c41a614738b9354d962" modify_external_catalogs: true
I20250115 14:51:52.149390 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_858dba41222b48a794cf0888977761c8" } new_table_name: "default.Table_8f6e7d6b890e4a449f170811c030b6f5" modify_external_catalogs: true
I20250115 14:51:52.151473 1165 tablet_replica.cc:331] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.153002 1165 raft_consensus.cc:2238] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.153225 1165 raft_consensus.cc:2267] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.153841 1164 tablet_bootstrap.cc:492] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.153896 1165 ts_tablet_manager.cc:1905] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.154084 1164 ts_tablet_manager.cc:1397] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.001s
I20250115 14:51:52.154336 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b4f282a7f918430ab5956c52b4e7f7fa" } modify_external_catalogs: true
I20250115 14:51:52.154538 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b4f282a7f918430ab5956c52b4e7f7fa" } modify_external_catalogs: true
I20250115 14:51:52.154888 1164 raft_consensus.cc:357] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.155094 1164 raft_consensus.cc:383] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.155169 1164 raft_consensus.cc:738] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.155398 1164 consensus_queue.cc:260] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.157624 1167 raft_consensus.cc:2238] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:52.157814 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_031dcfe40eda4ff49338ab79a0cf9844"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.157989 1167 pending_rounds.cc:62] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:52.158058 1167 pending_rounds.cc:69] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558308650651648 op_type: NO_OP noop_request { }
I20250115 14:51:52.158164 1167 raft_consensus.cc:2883] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:52.158265 1167 raft_consensus.cc:2267] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:52.158281 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_031dcfe40eda4ff49338ab79a0cf9844 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.158633 1164 ts_tablet_manager.cc:1428] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250115 14:51:52.161242 1164 tablet_bootstrap.cc:492] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.161378 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbf01a6c24b9449b8da3ee9428b7bd98" } modify_external_catalogs: true
I20250115 14:51:52.161518 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbf01a6c24b9449b8da3ee9428b7bd98" } modify_external_catalogs: true
I20250115 14:51:52.162763 1164 tablet_bootstrap.cc:654] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.165556 1167 ts_tablet_manager.cc:1905] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.166031 452 master_service.cc:946] ReplaceTablet: received request to replace tablet e115bdfa1d69421890122f9afdc23b29 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.167081 452 master_service.cc:946] ReplaceTablet: received request to replace tablet be18f1b6d8a24fabbe818811ef646c62 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.167770 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_47396588856f48a6ae9cb4d6469a39d1" } modify_external_catalogs: true
I20250115 14:51:52.167960 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_47396588856f48a6ae9cb4d6469a39d1" } modify_external_catalogs: true
I20250115 14:51:52.167217 1165 ts_tablet_manager.cc:1918] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.169646 1165 log.cc:1198] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f8346c87744f46daa623c414e7307299
I20250115 14:51:52.169514 753 tablet_service.cc:1467] Processing CreateTablet for tablet 24751e606ad94675b650d67a3738481e (DEFAULT_TABLE table=default.Table_49eb379e719f4677b13269d15ef7efad [id=af78a0b6494d4fd68224992dec097051]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.170006 1165 ts_tablet_manager.cc:1939] T f8346c87744f46daa623c414e7307299 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.170112 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24751e606ad94675b650d67a3738481e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.170513 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 379087c496d5401ebcbd1469ce1f68f3 (table default.Table_1d1bc2260e92451c99dfebfe09374e38 [id=959c2861880b40dcbb3780f3dff3b0f9]). Expected version 1 got 0
I20250115 14:51:52.170964 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f8346c87744f46daa623c414e7307299 (table default.Table_18add812314a4b0f97f9a61776eb768a [id=c85266bab3cd4586a91a82c1f7f96c09]) successfully deleted
I20250115 14:51:52.171248 1165 tablet_replica.cc:331] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.171429 1165 raft_consensus.cc:2238] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.171586 1165 raft_consensus.cc:2267] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.173542 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet f8346c87744f46daa623c414e7307299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.174075 1165 ts_tablet_manager.cc:1905] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.177767 1167 ts_tablet_manager.cc:1918] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:52.177884 1167 log.cc:1198] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/edd50ffca1ee49d0bbe96187480ea26b
I20250115 14:51:52.178249 1167 ts_tablet_manager.cc:1939] T edd50ffca1ee49d0bbe96187480ea26b P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.178282 1021 tablet_service.cc:1467] Processing CreateTablet for tablet bb792ca1b3f54def92f504ec8682d2c7 (DEFAULT_TABLE table=default.Table_733144094e85412998e706e7a0f93dbf [id=e1e34c5d139f4810a8f45a223e459e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.178625 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb792ca1b3f54def92f504ec8682d2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.179211 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet edd50ffca1ee49d0bbe96187480ea26b (table default.Table_ec6d2615d3f348d990f55956ac678e86 [id=f82deda2142247809c0f6a3fc2edf5c6]) successfully deleted
I20250115 14:51:52.183197 1165 ts_tablet_manager.cc:1918] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.186055 1165 log.cc:1198] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9bbd41d946d245b78e6686f8fae55449
I20250115 14:51:52.186240 888 tablet_service.cc:1467] Processing CreateTablet for tablet fad6f9f7f853418e85353edfd38cc80e (DEFAULT_TABLE table=default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.186607 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad6f9f7f853418e85353edfd38cc80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.186990 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0d566e954045fc85c4fbfe8052b050 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.187170 1165 ts_tablet_manager.cc:1939] T 9bbd41d946d245b78e6686f8fae55449 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.187844 756 tablet_service.cc:1467] Processing CreateTablet for tablet 9d3b4117171441a987ac043931fc6897 (DEFAULT_TABLE table=default.Table_8d37d2aba1094de3a9bc2eabee94643d [id=207a93a004da4fbc91db802c6939059a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.188211 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d3b4117171441a987ac043931fc6897. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.188647 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9bbd41d946d245b78e6686f8fae55449 (table default.Table_c128b7a02c834993a7068d76854faa7d [id=4745e3d5a5ba4176ad2b4f72b09846c5]) successfully deleted
I20250115 14:51:52.192595 754 tablet_service.cc:1467] Processing CreateTablet for tablet 69aed449a776493b8bdc7a31b47b62e8 (DEFAULT_TABLE table=default.Table_68b340d318b14c078d634b22c471d979 [id=e2d228ef5e814fd9bf50e2ed7e89be4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.192955 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69aed449a776493b8bdc7a31b47b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.193532 1165 tablet_replica.cc:331] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.193786 1165 raft_consensus.cc:2238] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:52.194077 1165 pending_rounds.cc:62] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:52.194175 1165 pending_rounds.cc:69] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558307390119936 op_type: NO_OP noop_request { }
I20250115 14:51:52.194358 1165 raft_consensus.cc:2883] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:52.194360 756 tablet_service.cc:1467] Processing CreateTablet for tablet bb792ca1b3f54def92f504ec8682d2c7 (DEFAULT_TABLE table=default.Table_733144094e85412998e706e7a0f93dbf [id=e1e34c5d139f4810a8f45a223e459e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.194494 1165 raft_consensus.cc:2267] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.194785 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb792ca1b3f54def92f504ec8682d2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.196118 1164 tablet_bootstrap.cc:492] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.196313 1164 ts_tablet_manager.cc:1397] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20250115 14:51:52.196427 1166 tablet_replica.cc:331] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.196622 1166 raft_consensus.cc:2238] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.196811 1166 raft_consensus.cc:2267] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.197094 1164 raft_consensus.cc:357] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.197314 1164 raft_consensus.cc:383] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.197404 1164 raft_consensus.cc:738] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.198079 1164 consensus_queue.cc:260] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:52.198216 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 715274a681f447c594e8fcd38d6fd609 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 715274a681f447c594e8fcd38d6fd609 already in progress: creating tablet
W20250115 14:51:52.198457 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 74bfda3861d148bc8b67ea071909d22d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74bfda3861d148bc8b67ea071909d22d already in progress: creating tablet
I20250115 14:51:52.198587 1165 ts_tablet_manager.cc:1905] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.199131 1166 ts_tablet_manager.cc:1905] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.200515 1164 ts_tablet_manager.cc:1428] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250115 14:51:52.200510 1207 raft_consensus.cc:491] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.200673 1207 raft_consensus.cc:513] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.201319 1207 leader_election.cc:290] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.201561 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 31d95c9359e54f1a88a07d5c5cce7fa1 (DEFAULT_TABLE table=default.Table_475c872b794349c397ee73627f500616 [id=db3a3335dd864c7384b2a414c31a6d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.201669 889 tablet_service.cc:1467] Processing CreateTablet for tablet 31d95c9359e54f1a88a07d5c5cce7fa1 (DEFAULT_TABLE table=default.Table_475c872b794349c397ee73627f500616 [id=db3a3335dd864c7384b2a414c31a6d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.201915 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d95c9359e54f1a88a07d5c5cce7fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.201983 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d95c9359e54f1a88a07d5c5cce7fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.203366 1164 tablet_bootstrap.cc:492] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.204576 1021 tablet_service.cc:1467] Processing CreateTablet for tablet fad6f9f7f853418e85353edfd38cc80e (DEFAULT_TABLE table=default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.204898 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad6f9f7f853418e85353edfd38cc80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.204898 1164 tablet_bootstrap.cc:654] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.205729 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4259dfb8ee4b618c0bacc910addd0a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:52.206468 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4259dfb8ee4b618c0bacc910addd0a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:52.206998 847 leader_election.cc:343] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.210005 1165 ts_tablet_manager.cc:1918] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:52.210124 1165 log.cc:1198] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/718f5f0300114292bf838f5f988ab566
I20250115 14:51:52.210448 1165 ts_tablet_manager.cc:1939] T 718f5f0300114292bf838f5f988ab566 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:51:52.211232 848 leader_election.cc:343] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.211301 1165 tablet_replica.cc:331] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.211369 848 leader_election.cc:304] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:52.211472 1165 raft_consensus.cc:2238] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.211571 1168 tablet_bootstrap.cc:492] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.211644 1165 raft_consensus.cc:2267] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.211671 1207 raft_consensus.cc:2743] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:52.211717 1168 ts_tablet_manager.cc:1397] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.079s user 0.014s sys 0.010s
I20250115 14:51:52.211936 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 718f5f0300114292bf838f5f988ab566 (table default.Table_40664b92268d4ba39f3f5852835f97d2 [id=da39f1615db74469824c3bf8eb74d260]) successfully deleted
I20250115 14:51:52.212459 1168 raft_consensus.cc:357] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.212636 1168 raft_consensus.cc:383] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.212719 1168 raft_consensus.cc:738] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.212982 1168 consensus_queue.cc:260] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.213522 1168 ts_tablet_manager.cc:1428] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:52.214107 1168 tablet_bootstrap.cc:492] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.215694 1168 tablet_bootstrap.cc:654] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.216356 754 tablet_service.cc:1467] Processing CreateTablet for tablet fad6f9f7f853418e85353edfd38cc80e (DEFAULT_TABLE table=default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.217144 741 tablet_service.cc:1467] Processing CreateTablet for tablet 31d95c9359e54f1a88a07d5c5cce7fa1 (DEFAULT_TABLE table=default.Table_475c872b794349c397ee73627f500616 [id=db3a3335dd864c7384b2a414c31a6d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.220501 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_68b340d318b14c078d634b22c471d979" } modify_external_catalogs: true
I20250115 14:51:52.220721 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_68b340d318b14c078d634b22c471d979" } modify_external_catalogs: true
I20250115 14:51:52.221693 1165 ts_tablet_manager.cc:1905] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.222543 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31d95c9359e54f1a88a07d5c5cce7fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.222268 451 catalog_manager.cc:5526] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:52.222719 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 379087c496d5401ebcbd1469ce1f68f3 (table default.Table_1d1bc2260e92451c99dfebfe09374e38 [id=959c2861880b40dcbb3780f3dff3b0f9]). Expected version 1 got 0
I20250115 14:51:52.232227 1165 ts_tablet_manager.cc:1918] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.232378 1165 log.cc:1198] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3ef5e4d75580465c9c0e36135503f332
I20250115 14:51:52.232712 1165 ts_tablet_manager.cc:1939] T 3ef5e4d75580465c9c0e36135503f332 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.234026 1166 ts_tablet_manager.cc:1918] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.234124 1165 tablet_replica.cc:331] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.234138 1166 log.cc:1198] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/1d0d566e954045fc85c4fbfe8052b050
I20250115 14:51:52.234338 1165 raft_consensus.cc:2238] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.234433 1166 ts_tablet_manager.cc:1939] T 1d0d566e954045fc85c4fbfe8052b050 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.234494 1165 raft_consensus.cc:2267] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.234515 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3ef5e4d75580465c9c0e36135503f332 (table default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1 [id=546100c082df4fb0984876cf4c60009c]) successfully deleted
W20250115 14:51:52.234752 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6326b1526191464485855b657e34a2b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6326b1526191464485855b657e34a2b9 already in progress: creating tablet
W20250115 14:51:52.234963 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 13c00f03edc74af8b084bb236487a347 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13c00f03edc74af8b084bb236487a347 already in progress: creating tablet
I20250115 14:51:52.235172 1164 tablet_bootstrap.cc:492] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
W20250115 14:51:52.235157 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e115bdfa1d69421890122f9afdc23b29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e115bdfa1d69421890122f9afdc23b29 already in progress: creating tablet
I20250115 14:51:52.235296 1164 ts_tablet_manager.cc:1397] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.004s
W20250115 14:51:52.235332 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c322301320ba425a94d7321c08f7f9e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c322301320ba425a94d7321c08f7f9e6 already in progress: creating tablet
W20250115 14:51:52.235563 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 8d43cd66b18f4649894a629637037822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d43cd66b18f4649894a629637037822 already in progress: creating tablet
I20250115 14:51:52.235836 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fad6f9f7f853418e85353edfd38cc80e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.235973 1164 raft_consensus.cc:357] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.236174 1164 raft_consensus.cc:383] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.236253 1164 raft_consensus.cc:738] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.236531 1164 consensus_queue.cc:260] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.238026 1164 ts_tablet_manager.cc:1428] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:52.238444 1164 tablet_bootstrap.cc:492] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.241111 1164 tablet_bootstrap.cc:654] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.243523 889 tablet_service.cc:1467] Processing CreateTablet for tablet f312f1e3a33a404ca7284c713859f156 (DEFAULT_TABLE table=default.Table_e1192ebeb25e40cbaf8670c6cf0d5f76 [id=1c779c8a53f54707abe1d619b30a912e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.243702 1165 ts_tablet_manager.cc:1905] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.244488 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f312f1e3a33a404ca7284c713859f156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.247598 1173 raft_consensus.cc:491] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.248649 1021 tablet_service.cc:1467] Processing CreateTablet for tablet f312f1e3a33a404ca7284c713859f156 (DEFAULT_TABLE table=default.Table_e1192ebeb25e40cbaf8670c6cf0d5f76 [id=1c779c8a53f54707abe1d619b30a912e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.248718 1173 raft_consensus.cc:513] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.249179 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f312f1e3a33a404ca7284c713859f156. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.249348 1173 leader_election.cc:290] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.250572 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 85ab1ccf420d41bd98cfda29caf5a619 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.252110 1049 raft_consensus.cc:1270] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:52.252940 1209 consensus_queue.cc:1035] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:51:52.257268 710 leader_election.cc:343] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 718f5f0300114292bf838f5f988ab566
W20250115 14:51:52.257465 710 leader_election.cc:343] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 718f5f0300114292bf838f5f988ab566
I20250115 14:51:52.257567 710 leader_election.cc:304] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:52.257808 1165 ts_tablet_manager.cc:1918] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.257928 1165 log.cc:1198] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d2529350d9264a1793c605ea9d5df9c6
I20250115 14:51:52.258141 1173 raft_consensus.cc:2743] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:52.258345 1165 ts_tablet_manager.cc:1939] T d2529350d9264a1793c605ea9d5df9c6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:51:52.258656 848 consensus_peers.cc:487] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:52.258949 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef5e4d75580465c9c0e36135503f332 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.259212 1165 tablet_replica.cc:331] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.259284 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d2529350d9264a1793c605ea9d5df9c6 (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]) successfully deleted
I20250115 14:51:52.259392 1165 raft_consensus.cc:2238] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.259682 1165 raft_consensus.cc:2267] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.260548 1168 tablet_bootstrap.cc:492] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.260715 1168 ts_tablet_manager.cc:1397] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.002s
I20250115 14:51:52.261451 1168 raft_consensus.cc:357] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.261664 1168 raft_consensus.cc:383] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.261744 1168 raft_consensus.cc:738] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.261859 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 718f5f0300114292bf838f5f988ab566 with delete_type TABLET_DATA_DELETED (replaced by tablet be18f1b6d8a24fabbe818811ef646c62) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.261994 1168 consensus_queue.cc:260] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.262459 1168 ts_tablet_manager.cc:1428] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:52.262717 1168 tablet_bootstrap.cc:492] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.262126 1166 tablet_replica.cc:331] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.263664 1166 raft_consensus.cc:2238] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.263880 1166 raft_consensus.cc:2267] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.265857 1168 tablet_bootstrap.cc:654] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.265986 1165 ts_tablet_manager.cc:1905] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.266602 1166 ts_tablet_manager.cc:1905] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.270382 1162 tablet_bootstrap.cc:492] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.270587 1162 ts_tablet_manager.cc:1397] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.000s
I20250115 14:51:52.271615 1162 raft_consensus.cc:357] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.273375 1162 raft_consensus.cc:383] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.273504 1162 raft_consensus.cc:738] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.273792 1162 consensus_queue.cc:260] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.274374 1162 ts_tablet_manager.cc:1428] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:51:52.275451 1162 tablet_bootstrap.cc:492] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.277361 1162 tablet_bootstrap.cc:654] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.278492 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 1d0d566e954045fc85c4fbfe8052b050 (table default.Table_fb13e7bf7ead4609a5d43f200ab36cad [id=a415575b540a4472bc26682438479572]) successfully deleted
I20250115 14:51:52.278927 754 tablet_service.cc:1467] Processing CreateTablet for tablet f312f1e3a33a404ca7284c713859f156 (DEFAULT_TABLE table=default.Table_e1192ebeb25e40cbaf8670c6cf0d5f76 [id=1c779c8a53f54707abe1d619b30a912e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.279829 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f312f1e3a33a404ca7284c713859f156. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:52.282014 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4161fce81834cd2ba4ffb789f6a5ba3 consensus state has no leader
I20250115 14:51:52.282405 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a75f913191b04255a02a30d5afea91c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.282919 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a75f913191b04255a02a30d5afea91c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:52.284082 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f312f1e3a33a404ca7284c713859f156 consensus state has no leader
I20250115 14:51:52.284336 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 70e2495496b746abb568e78377c83d1e from {username='slave'} at 127.0.0.1:47464
W20250115 14:51:52.285470 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6a9fb932c5b4545a4609d568f0e5494 consensus state has no leader
I20250115 14:51:52.285790 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1d1bc2260e92451c99dfebfe09374e38" } modify_external_catalogs: true
I20250115 14:51:52.285959 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1d1bc2260e92451c99dfebfe09374e38" } modify_external_catalogs: true
I20250115 14:51:52.286005 1271 tablet.cc:1721] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:52.286448 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet a7e59108a05540a0bca8cf133f31c525
I20250115 14:51:52.286640 446 master_service.cc:946] ReplaceTablet: received request to replace tablet f8346c87744f46daa623c414e7307299 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.287099 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b4259dfb8ee4b618c0bacc910addd0a from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.287564 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_843fb099548148f29e7b622a4ad34b61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.288074 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_843fb099548148f29e7b622a4ad34b61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.288771 1272 tablet.cc:1721] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:52.289479 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c156b9383c644939a6ada0bb7d7cf235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.289912 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c156b9383c644939a6ada0bb7d7cf235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.290180 1162 tablet_bootstrap.cc:492] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.290313 1162 ts_tablet_manager.cc:1397] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20250115 14:51:52.290971 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ad67354f35ce4564b88dbdffcabf3564
I20250115 14:51:52.291014 1265 mvcc.cc:204] Tried to move back new op lower bound from 7114558309373435904 to 7114558308160724992. Current Snapshot: MvccSnapshot[applied={T|T < 7114558309373435904}]
I20250115 14:51:52.291661 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c06f4c91e37e4b6f95521174e05efb24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.292096 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c06f4c91e37e4b6f95521174e05efb24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.292670 1166 ts_tablet_manager.cc:1918] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.292826 1166 log.cc:1198] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/85ab1ccf420d41bd98cfda29caf5a619
I20250115 14:51:52.293164 1166 ts_tablet_manager.cc:1939] T 85ab1ccf420d41bd98cfda29caf5a619 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.293984 1166 tablet_replica.cc:331] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.294183 756 tablet_service.cc:1514] Processing DeleteTablet for tablet ad67354f35ce4564b88dbdffcabf3564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.294010 1164 tablet_bootstrap.cc:492] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.294706 1164 ts_tablet_manager.cc:1397] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.006s
I20250115 14:51:52.295217 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 85ab1ccf420d41bd98cfda29caf5a619 (table default.Table_eb8583ed6e0c445b9883e8391d53b878 [id=af4d0a3d017c4e2cbc04fa416fa68e5a]) successfully deleted
I20250115 14:51:52.295517 1174 raft_consensus.cc:491] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.295758 889 tablet_service.cc:1514] Processing DeleteTablet for tablet ad67354f35ce4564b88dbdffcabf3564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.295667 1164 raft_consensus.cc:357] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.295678 1174 raft_consensus.cc:513] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.296510 1164 raft_consensus.cc:383] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:51:52.296509 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ad67354f35ce4564b88dbdffcabf3564 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad67354f35ce4564b88dbdffcabf3564 already in progress: creating tablet
I20250115 14:51:52.296593 1164 raft_consensus.cc:738] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.271848 1165 ts_tablet_manager.cc:1918] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.296936 1165 log.cc:1198] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/edd50ffca1ee49d0bbe96187480ea26b
I20250115 14:51:52.296954 1174 leader_election.cc:290] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.297174 1164 consensus_queue.cc:260] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.297286 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e115bdfa1d69421890122f9afdc23b29" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:52.297540 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e115bdfa1d69421890122f9afdc23b29" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:52.297734 1165 ts_tablet_manager.cc:1939] T edd50ffca1ee49d0bbe96187480ea26b P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:51:52.298132 847 leader_election.cc:343] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.298457 1166 raft_consensus.cc:2238] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.298615 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet edd50ffca1ee49d0bbe96187480ea26b (table default.Table_ec6d2615d3f348d990f55956ac678e86 [id=f82deda2142247809c0f6a3fc2edf5c6]) successfully deleted
I20250115 14:51:52.298779 1166 raft_consensus.cc:2267] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.298908 1165 tablet_replica.cc:331] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.299098 1165 raft_consensus.cc:2238] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:52.299320 1165 pending_rounds.cc:62] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:52.299405 1165 pending_rounds.cc:69] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558308253716480 op_type: NO_OP noop_request { }
I20250115 14:51:52.299517 1165 raft_consensus.cc:2883] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:52.299618 1165 raft_consensus.cc:2267] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:52.299743 848 leader_election.cc:343] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.299860 848 leader_election.cc:304] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:52.300112 1174 raft_consensus.cc:2743] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:52.300379 1165 ts_tablet_manager.cc:1905] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.300540 1164 ts_tablet_manager.cc:1428] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:51:52.301613 1164 tablet_bootstrap.cc:492] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.301900 1166 ts_tablet_manager.cc:1905] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.305963 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 81a5e53af38c4552ae3cbd953ce57abc
I20250115 14:51:52.306638 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a5e53af38c4552ae3cbd953ce57abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.307302 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 69aed449a776493b8bdc7a31b47b62e8
I20250115 14:51:52.307838 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a5e53af38c4552ae3cbd953ce57abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.307967 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6f92136d31774ef69d9893b440c2d545" } new_table_name: "default.Table_7281d9e98ae843dc915b4932d3d7d50b" modify_external_catalogs: true
I20250115 14:51:52.308105 1167 tablet_replica.cc:331] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.308380 1167 raft_consensus.cc:2238] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:52.308718 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aed449a776493b8bdc7a31b47b62e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.308799 1167 raft_consensus.cc:2267] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.309496 1164 tablet_bootstrap.cc:654] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.310789 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aed449a776493b8bdc7a31b47b62e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.311793 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_22fff0dd323f40378324128fc8bdddd8" } new_table_name: "default.Table_740738b002f34e2dba618cd146ff8dcd" modify_external_catalogs: true
I20250115 14:51:52.312436 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e20ad209d7044b09935691f002b9998 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:50 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.314013 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a5e53af38c4552ae3cbd953ce57abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.314234 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aed449a776493b8bdc7a31b47b62e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.314356 1165 ts_tablet_manager.cc:1918] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:52.314455 1165 log.cc:1198] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/85ab1ccf420d41bd98cfda29caf5a619
I20250115 14:51:52.314867 1165 ts_tablet_manager.cc:1939] T 85ab1ccf420d41bd98cfda29caf5a619 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.314888 1166 ts_tablet_manager.cc:1918] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.315004 1166 log.cc:1198] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/3ef5e4d75580465c9c0e36135503f332
I20250115 14:51:52.315327 1166 ts_tablet_manager.cc:1939] T 3ef5e4d75580465c9c0e36135503f332 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.315820 1162 raft_consensus.cc:357] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.316035 1162 raft_consensus.cc:383] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.316115 1162 raft_consensus.cc:738] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.316203 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 3ef5e4d75580465c9c0e36135503f332 (table default.Table_98ba6d7df8b74f9cbf6bc6875427e6e1 [id=546100c082df4fb0984876cf4c60009c]) successfully deleted
I20250115 14:51:52.316469 1167 ts_tablet_manager.cc:1905] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.316378 1162 consensus_queue.cc:260] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.317711 1162 ts_tablet_manager.cc:1428] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250115 14:51:52.317986 1162 tablet_bootstrap.cc:492] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.318339 1166 tablet_replica.cc:331] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.318521 1166 raft_consensus.cc:2238] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.318655 1166 raft_consensus.cc:2267] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.319392 1166 ts_tablet_manager.cc:1905] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.321197 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bf7d10b07fa646a8a235ec0eeb725f64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.321480 1162 tablet_bootstrap.cc:654] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:52.321702 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf7d10b07fa646a8a235ec0eeb725f64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.323241 1165 tablet_replica.cc:331] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.323426 1165 raft_consensus.cc:2238] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.323581 1165 raft_consensus.cc:2267] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.325845 1167 ts_tablet_manager.cc:1918] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:52.325979 1167 log.cc:1198] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/81a5e53af38c4552ae3cbd953ce57abc
I20250115 14:51:52.326352 1167 ts_tablet_manager.cc:1939] T 81a5e53af38c4552ae3cbd953ce57abc P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.329299 754 tablet_service.cc:1514] Processing DeleteTablet for tablet c5030b357a804d70b13a93b1d5d5fed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.329617 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 85ab1ccf420d41bd98cfda29caf5a619 (table default.Table_eb8583ed6e0c445b9883e8391d53b878 [id=af4d0a3d017c4e2cbc04fa416fa68e5a]) successfully deleted
W20250115 14:51:52.329893 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f8346c87744f46daa623c414e7307299 because the tablet was not found. No further retry: Not found: Tablet not found: f8346c87744f46daa623c414e7307299
I20250115 14:51:52.330224 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 81a5e53af38c4552ae3cbd953ce57abc (table default.Table_47396588856f48a6ae9cb4d6469a39d1 [id=9db6edcf4cfb491580916c914e61de0c]) successfully deleted
W20250115 14:51:52.330415 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 69aed449a776493b8bdc7a31b47b62e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69aed449a776493b8bdc7a31b47b62e8 already in progress: creating tablet
I20250115 14:51:52.332696 1164 tablet_bootstrap.cc:492] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.332847 1164 ts_tablet_manager.cc:1397] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250115 14:51:52.333561 1165 ts_tablet_manager.cc:1905] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.334786 1166 ts_tablet_manager.cc:1918] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.334928 1166 log.cc:1198] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/718f5f0300114292bf838f5f988ab566
I20250115 14:51:52.335490 1166 ts_tablet_manager.cc:1939] T 718f5f0300114292bf838f5f988ab566 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.336578 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 718f5f0300114292bf838f5f988ab566 (table default.Table_40664b92268d4ba39f3f5852835f97d2 [id=da39f1615db74469824c3bf8eb74d260]) successfully deleted
W20250115 14:51:52.337203 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ad67354f35ce4564b88dbdffcabf3564 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad67354f35ce4564b88dbdffcabf3564 already in progress: creating tablet
I20250115 14:51:52.337823 1166 tablet_replica.cc:331] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.337983 1166 raft_consensus.cc:2238] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.338135 1166 raft_consensus.cc:2267] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.338096 1164 raft_consensus.cc:357] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.338290 1164 raft_consensus.cc:383] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.338377 1164 raft_consensus.cc:738] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.338590 1164 consensus_queue.cc:260] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.339128 1166 ts_tablet_manager.cc:1905] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.340965 1165 ts_tablet_manager.cc:1918] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:52.341094 1165 log.cc:1198] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/81a5e53af38c4552ae3cbd953ce57abc
I20250115 14:51:52.341300 1164 ts_tablet_manager.cc:1428] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250115 14:51:52.341472 1165 ts_tablet_manager.cc:1939] T 81a5e53af38c4552ae3cbd953ce57abc P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.341708 1164 tablet_bootstrap.cc:492] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.342384 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 81a5e53af38c4552ae3cbd953ce57abc (table default.Table_47396588856f48a6ae9cb4d6469a39d1 [id=9db6edcf4cfb491580916c914e61de0c]) successfully deleted
I20250115 14:51:52.343182 912 raft_consensus.cc:1270] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:52.343319 775 raft_consensus.cc:3054] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:52.343902 1226 consensus_queue.cc:1035] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:52.344188 1164 tablet_bootstrap.cc:654] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.347859 775 raft_consensus.cc:1270] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:52.348495 1226 consensus_queue.cc:1035] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:51:52.352077 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 69aed449a776493b8bdc7a31b47b62e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69aed449a776493b8bdc7a31b47b62e8 already in progress: creating tablet
I20250115 14:51:52.356034 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 70e2495496b746abb568e78377c83d1e
I20250115 14:51:52.356550 447 catalog_manager.cc:6596] ReplaceTablet: tablet 70e2495496b746abb568e78377c83d1e deleted and replaced by tablet 4b90205ef15f470a99c013e7f2cf99c6
I20250115 14:51:52.357970 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e2495496b746abb568e78377c83d1e with delete_type TABLET_DATA_DELETED (replaced by tablet 4b90205ef15f470a99c013e7f2cf99c6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.361666 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e2495496b746abb568e78377c83d1e with delete_type TABLET_DATA_DELETED (replaced by tablet 4b90205ef15f470a99c013e7f2cf99c6) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.361960 1167 tablet_replica.cc:331] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.362134 1167 raft_consensus.cc:2238] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.362309 1167 raft_consensus.cc:2267] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.362509 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet ad67354f35ce4564b88dbdffcabf3564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.362946 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e2495496b746abb568e78377c83d1e with delete_type TABLET_DATA_DELETED (replaced by tablet 4b90205ef15f470a99c013e7f2cf99c6) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:52.363467 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ad67354f35ce4564b88dbdffcabf3564 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad67354f35ce4564b88dbdffcabf3564 already in progress: creating tablet
W20250115 14:51:52.364516 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 70e2495496b746abb568e78377c83d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70e2495496b746abb568e78377c83d1e already in progress: creating tablet
I20250115 14:51:52.368024 1167 ts_tablet_manager.cc:1905] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.373272 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_91244a7b0fa14291bcc2e9a2d776b4ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.374030 1167 ts_tablet_manager.cc:1918] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.374164 1167 log.cc:1198] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/70e2495496b746abb568e78377c83d1e
W20250115 14:51:52.374313 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91244a7b0fa14291bcc2e9a2d776b4ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.374529 1167 ts_tablet_manager.cc:1939] T 70e2495496b746abb568e78377c83d1e P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.375440 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 70e2495496b746abb568e78377c83d1e (table default.Table_2e17c919cc6f4992b7d7b764ba964949 [id=017270e9a3ad41f18c34b3cfc2c4cd42]) successfully deleted
I20250115 14:51:52.376492 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_edeb6add07054dcba5df71224ae8675c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.376981 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edeb6add07054dcba5df71224ae8675c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.380311 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_59fb977786f045519df5d218a7fd5e8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.380743 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59fb977786f045519df5d218a7fd5e8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.388197 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6674d45a98dc47b4b3c4abef46e52872"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.388726 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6674d45a98dc47b4b3c4abef46e52872 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.397392 1272 tablet.cc:1721] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:52.399317 1271 tablet.cc:1721] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:52.404565 1263 mvcc.cc:204] Tried to move back new op lower bound from 7114558309744599040 to 7114558308622684160. Current Snapshot: MvccSnapshot[applied={T|T < 7114558309744599040}]
W20250115 14:51:52.393368 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c56fe5b5b59b4017be3cb0ac877479b4 consensus state has no leader
I20250115 14:51:52.410044 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 9d3b4117171441a987ac043931fc6897 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.411630 449 master_service.cc:946] ReplaceTablet: received request to replace tablet f8346c87744f46daa623c414e7307299 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.415169 889 tablet_service.cc:1467] Processing CreateTablet for tablet fe0c1c829aff438a94bc41609bdb7841 (DEFAULT_TABLE table=default.Table_4b96cde69a5b411f97fe0f70ae0264c0 [id=a6994f260e1a4eada78d8c9e9fcf334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.415565 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe0c1c829aff438a94bc41609bdb7841. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:52.417523 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 66c6c392518a464dab664e5f9321f1f6 consensus state has no leader
I20250115 14:51:52.420936 1027 tablet_service.cc:1467] Processing CreateTablet for tablet fe0c1c829aff438a94bc41609bdb7841 (DEFAULT_TABLE table=default.Table_4b96cde69a5b411f97fe0f70ae0264c0 [id=a6994f260e1a4eada78d8c9e9fcf334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.421375 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe0c1c829aff438a94bc41609bdb7841. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.424613 1166 ts_tablet_manager.cc:1918] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.424902 1166 log.cc:1198] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/81a5e53af38c4552ae3cbd953ce57abc
I20250115 14:51:52.425275 1166 ts_tablet_manager.cc:1939] T 81a5e53af38c4552ae3cbd953ce57abc P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.427419 753 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd72a0d586d4a2c817f6a39daa07063 (DEFAULT_TABLE table=default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.427677 756 tablet_service.cc:1467] Processing CreateTablet for tablet ea0f104c1a71404591072e7a36b276b0 (DEFAULT_TABLE table=default.Table_c87fa8eb006845f5bf7b37b90f4c14c6 [id=717e74f1625e4a5a92c254f204df1629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.427809 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd72a0d586d4a2c817f6a39daa07063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.428043 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea0f104c1a71404591072e7a36b276b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.429991 1166 tablet_replica.cc:331] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.431536 1164 tablet_bootstrap.cc:492] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.431744 1164 ts_tablet_manager.cc:1397] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.001s
I20250115 14:51:52.431825 741 tablet_service.cc:1467] Processing CreateTablet for tablet 2711a7d437f44b2fa11504d8e400e360 (DEFAULT_TABLE table=default.Table_86166d8e2b8e40f59fc59eb5d8076fbc [id=a02ee91f154e41fd9ee54b476fd77331]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.432145 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2711a7d437f44b2fa11504d8e400e360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.433019 1164 raft_consensus.cc:357] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.433295 1164 raft_consensus.cc:383] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.433382 1164 raft_consensus.cc:738] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.433637 1164 consensus_queue.cc:260] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.434458 1164 ts_tablet_manager.cc:1428] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:51:52.430354 1166 raft_consensus.cc:2238] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.434741 1166 raft_consensus.cc:2267] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.435030 751 tablet_service.cc:1467] Processing CreateTablet for tablet 58cca6072e4e4f608c6a45be279304f8 (DEFAULT_TABLE table=default.Table_ca27ab47d077491489383629eaaf1437 [id=5ec280e869eb4ba6bec54fbc22af1409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.435341 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58cca6072e4e4f608c6a45be279304f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.435501 1166 ts_tablet_manager.cc:1905] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.434676 749 tablet_service.cc:1467] Processing CreateTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b (DEFAULT_TABLE table=default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.440093 1164 tablet_bootstrap.cc:492] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.440106 747 tablet_service.cc:1467] Processing CreateTablet for tablet b027233cfd314d0ebda472b4588d3f00 (DEFAULT_TABLE table=default.Table_dfdad82a801948d08483f75e70f61100 [id=a5dc62c04f1747128d36f3baf75a6fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.441001 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 2711a7d437f44b2fa11504d8e400e360 (DEFAULT_TABLE table=default.Table_86166d8e2b8e40f59fc59eb5d8076fbc [id=a02ee91f154e41fd9ee54b476fd77331]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.441146 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 58cca6072e4e4f608c6a45be279304f8 (DEFAULT_TABLE table=default.Table_ca27ab47d077491489383629eaaf1437 [id=5ec280e869eb4ba6bec54fbc22af1409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.441211 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd72a0d586d4a2c817f6a39daa07063 (DEFAULT_TABLE table=default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.441318 1162 tablet_bootstrap.cc:492] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.441356 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2711a7d437f44b2fa11504d8e400e360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.441485 1162 ts_tablet_manager.cc:1397] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.124s user 0.021s sys 0.044s
I20250115 14:51:52.441545 1018 tablet_service.cc:1467] Processing CreateTablet for tablet b027233cfd314d0ebda472b4588d3f00 (DEFAULT_TABLE table=default.Table_dfdad82a801948d08483f75e70f61100 [id=a5dc62c04f1747128d36f3baf75a6fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.441553 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58cca6072e4e4f608c6a45be279304f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.441879 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b027233cfd314d0ebda472b4588d3f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.440107 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1fa787e364d4ca6a4b756a8f4beba3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.442328 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd72a0d586d4a2c817f6a39daa07063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.442353 1162 raft_consensus.cc:357] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.442572 1162 raft_consensus.cc:383] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.442644 1162 raft_consensus.cc:738] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.442977 1162 consensus_queue.cc:260] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.443528 1162 ts_tablet_manager.cc:1428] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:52.443897 1162 tablet_bootstrap.cc:492] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.444614 1019 tablet_service.cc:1467] Processing CreateTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b (DEFAULT_TABLE table=default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.444949 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1fa787e364d4ca6a4b756a8f4beba3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.445559 1162 tablet_bootstrap.cc:654] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.448329 885 tablet_service.cc:1467] Processing CreateTablet for tablet b027233cfd314d0ebda472b4588d3f00 (DEFAULT_TABLE table=default.Table_dfdad82a801948d08483f75e70f61100 [id=a5dc62c04f1747128d36f3baf75a6fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.448357 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd72a0d586d4a2c817f6a39daa07063 (DEFAULT_TABLE table=default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.448741 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b027233cfd314d0ebda472b4588d3f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.449018 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd72a0d586d4a2c817f6a39daa07063. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.449299 888 tablet_service.cc:1467] Processing CreateTablet for tablet ea0f104c1a71404591072e7a36b276b0 (DEFAULT_TABLE table=default.Table_c87fa8eb006845f5bf7b37b90f4c14c6 [id=717e74f1625e4a5a92c254f204df1629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.449520 890 tablet_service.cc:1467] Processing CreateTablet for tablet 58cca6072e4e4f608c6a45be279304f8 (DEFAULT_TABLE table=default.Table_ca27ab47d077491489383629eaaf1437 [id=5ec280e869eb4ba6bec54fbc22af1409]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.449616 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea0f104c1a71404591072e7a36b276b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.449884 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58cca6072e4e4f608c6a45be279304f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.450223 879 tablet_service.cc:1467] Processing CreateTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b (DEFAULT_TABLE table=default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.450666 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1fa787e364d4ca6a4b756a8f4beba3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.452418 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbd41d946d245b78e6686f8fae55449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.452649 887 tablet_service.cc:1514] Processing DeleteTablet for tablet e115bdfa1d69421890122f9afdc23b29 with delete_type TABLET_DATA_DELETED (replaced by tablet f312f1e3a33a404ca7284c713859f156) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.452833 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f8346c87744f46daa623c414e7307299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.453027 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4259dfb8ee4b618c0bacc910addd0a with delete_type TABLET_DATA_DELETED (replaced by tablet 3a38a6bc32314396b497e7ae166a63f7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.453202 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e2495496b746abb568e78377c83d1e with delete_type TABLET_DATA_DELETED (replaced by tablet 4b90205ef15f470a99c013e7f2cf99c6) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.453406 882 tablet_service.cc:1514] Processing DeleteTablet for tablet d2529350d9264a1793c605ea9d5df9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fad6f9f7f853418e85353edfd38cc80e) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.453590 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 81a5e53af38c4552ae3cbd953ce57abc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.453836 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 715274a681f447c594e8fcd38d6fd609 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.456142 884 tablet_service.cc:1467] Processing CreateTablet for tablet 2711a7d437f44b2fa11504d8e400e360 (DEFAULT_TABLE table=default.Table_86166d8e2b8e40f59fc59eb5d8076fbc [id=a02ee91f154e41fd9ee54b476fd77331]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.456521 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2711a7d437f44b2fa11504d8e400e360. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.458159 1167 tablet_replica.cc:331] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.458408 1167 raft_consensus.cc:2238] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.458596 1167 raft_consensus.cc:2267] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.459497 1167 ts_tablet_manager.cc:1905] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.464574 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 81a5e53af38c4552ae3cbd953ce57abc (table default.Table_47396588856f48a6ae9cb4d6469a39d1 [id=9db6edcf4cfb491580916c914e61de0c]) successfully deleted
W20250115 14:51:52.464869 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 69aed449a776493b8bdc7a31b47b62e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69aed449a776493b8bdc7a31b47b62e8 already in progress: creating tablet
I20250115 14:51:52.467515 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 379087c496d5401ebcbd1469ce1f68f3
I20250115 14:51:52.478353 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 379087c496d5401ebcbd1469ce1f68f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.478757 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9d3b4117171441a987ac043931fc6897
I20250115 14:51:52.479240 452 catalog_manager.cc:6596] ReplaceTablet: tablet 9d3b4117171441a987ac043931fc6897 deleted and replaced by tablet 906fd4d0aed1497ebe0c7316aa6424bb
I20250115 14:51:52.484613 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b027233cfd314d0ebda472b4588d3f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.486663 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 379087c496d5401ebcbd1469ce1f68f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.486809 1021 tablet_service.cc:1467] Processing CreateTablet for tablet ea0f104c1a71404591072e7a36b276b0 (DEFAULT_TABLE table=default.Table_c87fa8eb006845f5bf7b37b90f4c14c6 [id=717e74f1625e4a5a92c254f204df1629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.488823 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d3b4117171441a987ac043931fc6897 with delete_type TABLET_DATA_DELETED (replaced by tablet 906fd4d0aed1497ebe0c7316aa6424bb) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.490383 1173 raft_consensus.cc:491] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.493376 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet d2529350d9264a1793c605ea9d5df9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fad6f9f7f853418e85353edfd38cc80e) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.496028 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 379087c496d5401ebcbd1469ce1f68f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:52.497058 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d2529350d9264a1793c605ea9d5df9c6 because the tablet was not found. No further retry: Not found: Tablet not found: d2529350d9264a1793c605ea9d5df9c6
I20250115 14:51:52.497292 1173 raft_consensus.cc:513] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.497956 1173 leader_election.cc:290] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.498091 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_319faebd004b4a1e93821d947ef288d0" } new_table_name: "default.Table_de1856a934f84a1fa118e04a42ef373e" modify_external_catalogs: true
I20250115 14:51:52.502530 1167 ts_tablet_manager.cc:1918] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.503139 1167 log.cc:1198] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9bbd41d946d245b78e6686f8fae55449
I20250115 14:51:52.503610 1167 ts_tablet_manager.cc:1939] T 9bbd41d946d245b78e6686f8fae55449 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.504205 1167 tablet_replica.cc:331] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.504477 1167 raft_consensus.cc:2238] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.504717 1167 raft_consensus.cc:2267] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.506278 1167 ts_tablet_manager.cc:1905] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.505465 1168 tablet_bootstrap.cc:492] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.507225 1168 ts_tablet_manager.cc:1397] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.245s user 0.006s sys 0.001s
I20250115 14:51:52.509459 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a75f913191b04255a02a30d5afea91c5" } new_table_name: "default.Table_18aff4b27c7846fbaede10b093240587" modify_external_catalogs: true
I20250115 14:51:52.509546 1168 raft_consensus.cc:357] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.509922 1168 raft_consensus.cc:383] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.510048 1168 raft_consensus.cc:738] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.511201 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 379087c496d5401ebcbd1469ce1f68f3 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.511500 1168 consensus_queue.cc:260] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.512023 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2a2f7c950304406484083d56f0b3177e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.512596 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_86166d8e2b8e40f59fc59eb5d8076fbc" } new_table_name: "default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8" modify_external_catalogs: true
I20250115 14:51:52.513221 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7281d9e98ae843dc915b4932d3d7d50b" } modify_external_catalogs: true
I20250115 14:51:52.513375 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7281d9e98ae843dc915b4932d3d7d50b" } modify_external_catalogs: true
I20250115 14:51:52.514127 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8d37d2aba1094de3a9bc2eabee94643d" } modify_external_catalogs: true
I20250115 14:51:52.514950 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8d37d2aba1094de3a9bc2eabee94643d" } modify_external_catalogs: true
I20250115 14:51:52.515887 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_031dcfe40eda4ff49338ab79a0cf9844" } modify_external_catalogs: true
I20250115 14:51:52.516045 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_031dcfe40eda4ff49338ab79a0cf9844" } modify_external_catalogs: true
W20250115 14:51:52.517314 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a2f7c950304406484083d56f0b3177e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.518041 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_740738b002f34e2dba618cd146ff8dcd" } modify_external_catalogs: true
I20250115 14:51:52.518206 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_740738b002f34e2dba618cd146ff8dcd" } modify_external_catalogs: true
I20250115 14:51:52.519979 1168 ts_tablet_manager.cc:1428] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250115 14:51:52.520848 1168 tablet_bootstrap.cc:492] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.522718 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9bbd41d946d245b78e6686f8fae55449 (table default.Table_c128b7a02c834993a7068d76854faa7d [id=4745e3d5a5ba4176ad2b4f72b09846c5]) successfully deleted
I20250115 14:51:52.526552 1165 tablet_replica.cc:331] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.526775 1165 raft_consensus.cc:2238] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:52.527124 1165 raft_consensus.cc:2267] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.528151 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d3b4117171441a987ac043931fc6897 with delete_type TABLET_DATA_DELETED (replaced by tablet 906fd4d0aed1497ebe0c7316aa6424bb) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.528640 1166 ts_tablet_manager.cc:1918] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.528812 1166 log.cc:1198] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/0e20ad209d7044b09935691f002b9998
I20250115 14:51:52.529168 1166 ts_tablet_manager.cc:1939] T 0e20ad209d7044b09935691f002b9998 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:51:52.530148 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf909c84eb314d95b73b6de82aebe9f2 consensus state has no leader
I20250115 14:51:52.530483 754 tablet_service.cc:1467] Processing CreateTablet for tablet fe0c1c829aff438a94bc41609bdb7841 (DEFAULT_TABLE table=default.Table_4b96cde69a5b411f97fe0f70ae0264c0 [id=a6994f260e1a4eada78d8c9e9fcf334e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:52.531507 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2711a7d437f44b2fa11504d8e400e360 consensus state has no leader
I20250115 14:51:52.531757 447 master_service.cc:946] ReplaceTablet: received request to replace tablet c322301320ba425a94d7321c08f7f9e6 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.536041 889 tablet_service.cc:1514] Processing DeleteTablet for tablet be18f1b6d8a24fabbe818811ef646c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:52.539217 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab1a38186c5042ff851f57979ab80dbc has no consensus state
I20250115 14:51:52.539554 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 906fd4d0aed1497ebe0c7316aa6424bb
I20250115 14:51:52.540751 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1653219949dc4fdeb257d0ba8c88c231"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.541316 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1653219949dc4fdeb257d0ba8c88c231 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.541721 1168 tablet_bootstrap.cc:654] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.545629 1165 ts_tablet_manager.cc:1905] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.546501 1167 ts_tablet_manager.cc:1918] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.546564 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea0f104c1a71404591072e7a36b276b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.546629 1167 log.cc:1198] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e115bdfa1d69421890122f9afdc23b29
I20250115 14:51:52.546798 1162 tablet_bootstrap.cc:492] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.546940 1162 ts_tablet_manager.cc:1397] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.055s
I20250115 14:51:52.547000 1167 ts_tablet_manager.cc:1939] T e115bdfa1d69421890122f9afdc23b29 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.547791 1162 raft_consensus.cc:357] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.547928 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet e115bdfa1d69421890122f9afdc23b29 (table default.Table_2752e5f3a4ca4c41a614738b9354d962 [id=1c779c8a53f54707abe1d619b30a912e]) successfully deleted
I20250115 14:51:52.548017 1162 raft_consensus.cc:383] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.548110 1162 raft_consensus.cc:738] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.548244 1167 tablet_replica.cc:331] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.548439 1167 raft_consensus.cc:2238] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.548350 1162 consensus_queue.cc:260] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.548588 1167 raft_consensus.cc:2267] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.549099 1167 ts_tablet_manager.cc:1905] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.549772 1162 ts_tablet_manager.cc:1428] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:52.550050 1162 tablet_bootstrap.cc:492] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.551687 1162 tablet_bootstrap.cc:654] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.552760 1168 tablet_bootstrap.cc:492] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.552912 1168 ts_tablet_manager.cc:1397] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20250115 14:51:52.553642 1168 raft_consensus.cc:357] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.553931 1168 raft_consensus.cc:383] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.554003 1168 raft_consensus.cc:738] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.554212 1168 consensus_queue.cc:260] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.555032 1168 ts_tablet_manager.cc:1428] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:52.555730 1168 tablet_bootstrap.cc:492] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:51:52.557471 710 leader_election.cc:343] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: c5030b357a804d70b13a93b1d5d5fed1
I20250115 14:51:52.557657 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 0e20ad209d7044b09935691f002b9998 (table default.Table_5d2e74e77f87456396aed66d4d21f840 [id=1fc8e792fbbe4faf85699a5a99b99908]) successfully deleted
I20250115 14:51:52.558964 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe0c1c829aff438a94bc41609bdb7841. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.564560 1162 tablet_bootstrap.cc:492] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.564796 1162 ts_tablet_manager.cc:1397] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.001s
I20250115 14:51:52.565729 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 13c00f03edc74af8b084bb236487a347 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.565853 1162 raft_consensus.cc:357] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.566068 1162 raft_consensus.cc:383] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.566134 1162 raft_consensus.cc:738] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.566581 1162 consensus_queue.cc:260] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.567544 1174 raft_consensus.cc:491] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.567715 1174 raft_consensus.cc:513] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.568217 1162 ts_tablet_manager.cc:1428] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:52.568380 1174 leader_election.cc:290] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.568840 1162 tablet_bootstrap.cc:492] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.569027 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715274a681f447c594e8fcd38d6fd609" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:52.569237 1049 raft_consensus.cc:2463] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:52.569653 847 leader_election.cc:304] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:52.569938 1174 raft_consensus.cc:2798] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:52.570542 1174 raft_consensus.cc:491] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:52.570664 1174 raft_consensus.cc:3054] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:52.571171 710 leader_election.cc:343] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: c5030b357a804d70b13a93b1d5d5fed1
I20250115 14:51:52.571810 1166 tablet_replica.cc:331] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.572845 1162 tablet_bootstrap.cc:654] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.573297 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 66c6c392518a464dab664e5f9321f1f6 (table default.Table_7281d9e98ae843dc915b4932d3d7d50b [id=52b6d8da69f74139a394cafe66e48a35]). Expected version 1 got 0
I20250115 14:51:52.573269 1174 raft_consensus.cc:513] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.574532 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715274a681f447c594e8fcd38d6fd609" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:52.574702 1049 raft_consensus.cc:3054] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:52.574733 1164 tablet_bootstrap.cc:654] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.571343 710 leader_election.cc:304] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:52.575729 1168 tablet_bootstrap.cc:654] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.577283 1049 raft_consensus.cc:2463] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:52.577840 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d3b4117171441a987ac043931fc6897 with delete_type TABLET_DATA_DELETED (replaced by tablet 906fd4d0aed1497ebe0c7316aa6424bb) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.577960 847 leader_election.cc:304] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:52.578121 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]). Expected version 2 got 0
I20250115 14:51:52.578229 1207 raft_consensus.cc:2798] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:52.578354 1207 raft_consensus.cc:695] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:52.578593 1207 consensus_queue.cc:237] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:52.578720 447 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet bd6d837c64ad4503a224ced67ff1d053 consensus state has no leader
I20250115 14:51:52.579348 1166 raft_consensus.cc:2238] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.580672 1173 raft_consensus.cc:2701] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Received pre-election callback for term 1 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20250115 14:51:52.581544 1020 tablet_service.cc:1467] Processing CreateTablet for tablet a6b76cb343fb4456ac07fd3c6e9888de (DEFAULT_TABLE table=default.Table_91244a7b0fa14291bcc2e9a2d776b4ae [id=4a31d55ea43e4ce5bbda5b78fdbfffa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.581981 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6b76cb343fb4456ac07fd3c6e9888de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.584395 1166 raft_consensus.cc:2267] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.585353 1167 ts_tablet_manager.cc:1918] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.585491 1167 log.cc:1198] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f8346c87744f46daa623c414e7307299
I20250115 14:51:52.585893 1167 ts_tablet_manager.cc:1939] T f8346c87744f46daa623c414e7307299 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.585793 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715274a681f447c594e8fcd38d6fd609" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:52.586356 848 leader_election.cc:343] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.586942 1166 ts_tablet_manager.cc:1905] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.588057 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f8346c87744f46daa623c414e7307299 (table default.Table_18add812314a4b0f97f9a61776eb768a [id=c85266bab3cd4586a91a82c1f7f96c09]) successfully deleted
I20250115 14:51:52.589093 1174 leader_election.cc:290] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.589370 1167 tablet_replica.cc:331] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.589552 1167 raft_consensus.cc:2238] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.589736 1167 raft_consensus.cc:2267] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.590231 1167 ts_tablet_manager.cc:1905] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.594045 1165 ts_tablet_manager.cc:1918] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:52.594527 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fa872457db7140b6a13bfc998e32581b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.595150 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa872457db7140b6a13bfc998e32581b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.595227 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "715274a681f447c594e8fcd38d6fd609" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:52.595665 848 leader_election.cc:343] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.595824 753 tablet_service.cc:1467] Processing CreateTablet for tablet a6b76cb343fb4456ac07fd3c6e9888de (DEFAULT_TABLE table=default.Table_91244a7b0fa14291bcc2e9a2d776b4ae [id=4a31d55ea43e4ce5bbda5b78fdbfffa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.596212 1168 tablet_bootstrap.cc:492] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.596349 1168 ts_tablet_manager.cc:1397] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.003s
I20250115 14:51:52.596541 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6b76cb343fb4456ac07fd3c6e9888de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.597069 1168 raft_consensus.cc:357] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.597249 1168 raft_consensus.cc:383] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.597340 1168 raft_consensus.cc:738] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.597574 1168 consensus_queue.cc:260] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.598055 1168 ts_tablet_manager.cc:1428] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:52.598239 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1f1b7f31ef374e49add057beb69dbbcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.598330 1168 tablet_bootstrap.cc:492] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:51:52.599171 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f1b7f31ef374e49add057beb69dbbcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.600926 1168 tablet_bootstrap.cc:654] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.602049 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_461138269c764aceb6cd3baa42a84638"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.602538 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_461138269c764aceb6cd3baa42a84638 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.605382 1167 ts_tablet_manager.cc:1918] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.605499 1167 log.cc:1198] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/2b4259dfb8ee4b618c0bacc910addd0a
I20250115 14:51:52.605857 1167 ts_tablet_manager.cc:1939] T 2b4259dfb8ee4b618c0bacc910addd0a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.606835 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 2b4259dfb8ee4b618c0bacc910addd0a (table default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]) successfully deleted
I20250115 14:51:52.607818 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9502b06eea13418c8ce173b8f4b5ec08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.609014 751 tablet_service.cc:1514] Processing DeleteTablet for tablet edd50ffca1ee49d0bbe96187480ea26b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.609210 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]). Expected version 2 got 0
W20250115 14:51:52.610316 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9502b06eea13418c8ce173b8f4b5ec08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.612051 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dfdad82a801948d08483f75e70f61100" } new_table_name: "default.Table_f7332a834b6a46018bacf978fc59830b" modify_external_catalogs: true
W20250115 14:51:52.612138 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 70e2495496b746abb568e78377c83d1e because the tablet was not found. No further retry: Not found: Tablet not found: 70e2495496b746abb568e78377c83d1e
I20250115 14:51:52.612828 1167 tablet_replica.cc:331] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.612846 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 22edaa6e70d942ba98777973476f449d (table default.Table_1b3a658f6eda417cb956eba9cdc5a2af [id=d7ba4cda40964e1d839efe9e0f1615c0]). Expected version 3 got 0
I20250115 14:51:52.613039 1167 raft_consensus.cc:2238] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.613210 1167 raft_consensus.cc:2267] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.613770 1167 ts_tablet_manager.cc:1905] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.613958 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0826262d07e14081b5a0ab88d9d4a20f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.614619 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0826262d07e14081b5a0ab88d9d4a20f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.619282 1168 tablet_bootstrap.cc:492] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.619436 1168 ts_tablet_manager.cc:1397] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20250115 14:51:52.620180 1168 raft_consensus.cc:357] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.620410 1168 raft_consensus.cc:383] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.620488 1168 raft_consensus.cc:738] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.620721 1168 consensus_queue.cc:260] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.621513 889 tablet_service.cc:1467] Processing CreateTablet for tablet a6b76cb343fb4456ac07fd3c6e9888de (DEFAULT_TABLE table=default.Table_91244a7b0fa14291bcc2e9a2d776b4ae [id=4a31d55ea43e4ce5bbda5b78fdbfffa6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.621892 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6b76cb343fb4456ac07fd3c6e9888de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.623387 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 66c6c392518a464dab664e5f9321f1f6
I20250115 14:51:52.624454 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6c392518a464dab664e5f9321f1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.628412 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6c392518a464dab664e5f9321f1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.628779 1166 ts_tablet_manager.cc:1918] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.628887 1166 log.cc:1198] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/c5030b357a804d70b13a93b1d5d5fed1
I20250115 14:51:52.621201 1168 ts_tablet_manager.cc:1428] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:52.629545 1166 ts_tablet_manager.cc:1939] T c5030b357a804d70b13a93b1d5d5fed1 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.631860 1168 tablet_bootstrap.cc:492] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.631844 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet c5030b357a804d70b13a93b1d5d5fed1 (table default.Table_05cc51b3f5ba4d518525c47cbfba4527 [id=7fc50307921343ddb6368d736c8adbfc]) successfully deleted
W20250115 14:51:52.633826 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 70e2495496b746abb568e78377c83d1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70e2495496b746abb568e78377c83d1e already in progress: creating tablet
I20250115 14:51:52.634121 1166 tablet_replica.cc:331] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.634294 1166 raft_consensus.cc:2238] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.634426 1166 pending_rounds.cc:62] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 2 pending ops.
I20250115 14:51:52.634635 1166 pending_rounds.cc:69] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7114558309744599040 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "379087c496d5401ebcbd1469ce1f68f3" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_1d1bc2260e92451c99dfebfe09374e38" dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" new_extra_config { } }
I20250115 14:51:52.635311 1166 pending_rounds.cc:69] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558308622684160 op_type: NO_OP noop_request { }
I20250115 14:51:52.635469 1166 raft_consensus.cc:2883] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:52.635599 1166 raft_consensus.cc:2267] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.636622 1168 tablet_bootstrap.cc:654] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.636884 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 745153c106844076bb020f9ed026c97a
I20250115 14:51:52.638031 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 745153c106844076bb020f9ed026c97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.638278 1166 ts_tablet_manager.cc:1905] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.639297 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c56fe5b5b59b4017be3cb0ac877479b4
I20250115 14:51:52.640581 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 745153c106844076bb020f9ed026c97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.641661 753 tablet_service.cc:1514] Processing DeleteTablet for tablet c56fe5b5b59b4017be3cb0ac877479b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.644131 889 tablet_service.cc:1514] Processing DeleteTablet for tablet c56fe5b5b59b4017be3cb0ac877479b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.645350 1167 ts_tablet_manager.cc:1918] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.645367 1166 ts_tablet_manager.cc:1918] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:52.645493 1167 log.cc:1198] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d2529350d9264a1793c605ea9d5df9c6
I20250115 14:51:52.645624 1166 log.cc:1198] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/379087c496d5401ebcbd1469ce1f68f3
I20250115 14:51:52.646044 1166 ts_tablet_manager.cc:1939] T 379087c496d5401ebcbd1469ce1f68f3 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.646847 1167 ts_tablet_manager.cc:1939] T d2529350d9264a1793c605ea9d5df9c6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.647230 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 379087c496d5401ebcbd1469ce1f68f3 (table default.Table_1d1bc2260e92451c99dfebfe09374e38 [id=959c2861880b40dcbb3780f3dff3b0f9]) successfully deleted
I20250115 14:51:52.649490 1166 tablet_replica.cc:331] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.649669 1166 raft_consensus.cc:2238] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.649857 1166 raft_consensus.cc:2267] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.652315 1166 ts_tablet_manager.cc:1905] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.653218 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f7c3d078ce8a49cfa5965cb29b102e42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.653823 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7c3d078ce8a49cfa5965cb29b102e42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.655431 889 tablet_service.cc:1514] Processing DeleteTablet for tablet c322301320ba425a94d7321c08f7f9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.655517 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bf7d10b07fa646a8a235ec0eeb725f64" } new_table_name: "default.Table_721928c380b840ad84f9f5030b52632f" modify_external_catalogs: true
I20250115 14:51:52.656424 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d2529350d9264a1793c605ea9d5df9c6 (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]) successfully deleted
I20250115 14:51:52.656519 1167 tablet_replica.cc:331] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.656692 1167 raft_consensus.cc:2238] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
W20250115 14:51:52.656749 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 81a5e53af38c4552ae3cbd953ce57abc because the tablet was not found. No further retry: Not found: Tablet not found: 81a5e53af38c4552ae3cbd953ce57abc
I20250115 14:51:52.656927 1167 pending_rounds.cc:62] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:52.656981 1167 pending_rounds.cc:69] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558310724403200 op_type: NO_OP noop_request { }
I20250115 14:51:52.657107 1167 raft_consensus.cc:2883] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:52.657197 1167 raft_consensus.cc:2267] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.659189 454 master_service.cc:946] ReplaceTablet: received request to replace tablet f224f857b1644596853acd9e3a2a4a5f from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.659420 1166 ts_tablet_manager.cc:1918] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.659507 1166 log.cc:1198] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/edd50ffca1ee49d0bbe96187480ea26b
I20250115 14:51:52.659776 1166 ts_tablet_manager.cc:1939] T edd50ffca1ee49d0bbe96187480ea26b P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.660579 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet edd50ffca1ee49d0bbe96187480ea26b (table default.Table_ec6d2615d3f348d990f55956ac678e86 [id=f82deda2142247809c0f6a3fc2edf5c6]) successfully deleted
I20250115 14:51:52.660977 1167 ts_tablet_manager.cc:1905] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:52.661080 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 66c6c392518a464dab664e5f9321f1f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 66c6c392518a464dab664e5f9321f1f6 already in progress: creating tablet
W20250115 14:51:52.661283 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 745153c106844076bb020f9ed026c97a because the tablet was not found. No further retry: Not found: Tablet not found: 745153c106844076bb020f9ed026c97a
W20250115 14:51:52.661475 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c56fe5b5b59b4017be3cb0ac877479b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c56fe5b5b59b4017be3cb0ac877479b4 already in progress: creating tablet
I20250115 14:51:52.661958 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b17b7ebacc9a4d32b756eff66de640dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.662354 1168 tablet_bootstrap.cc:492] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.662492 1168 ts_tablet_manager.cc:1397] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
W20250115 14:51:52.662482 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b17b7ebacc9a4d32b756eff66de640dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.663168 1168 raft_consensus.cc:357] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.663327 1168 raft_consensus.cc:383] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.663412 1168 raft_consensus.cc:738] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.663641 1168 consensus_queue.cc:260] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.664914 1168 ts_tablet_manager.cc:1428] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:52.665181 1168 tablet_bootstrap.cc:492] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.666751 1168 tablet_bootstrap.cc:654] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.669831 1168 tablet_bootstrap.cc:492] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.669973 1168 ts_tablet_manager.cc:1397] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250115 14:51:52.670491 1168 raft_consensus.cc:357] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.670661 1168 raft_consensus.cc:383] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.670740 1168 raft_consensus.cc:738] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.670958 1168 consensus_queue.cc:260] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.671499 1168 ts_tablet_manager.cc:1428] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250115 14:51:52.672376 1168 tablet_bootstrap.cc:492] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.673601 753 tablet_service.cc:1467] Processing CreateTablet for tablet 4821224ac8f1472d8b1e8afc06a118a3 (DEFAULT_TABLE table=default.Table_bf7d10b07fa646a8a235ec0eeb725f64 [id=c5f831177088497dbf1fee130e231e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.673991 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4821224ac8f1472d8b1e8afc06a118a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.674022 1168 tablet_bootstrap.cc:654] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.674916 1167 ts_tablet_manager.cc:1918] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:52.675065 1167 log.cc:1198] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/715274a681f447c594e8fcd38d6fd609
I20250115 14:51:52.675418 1167 ts_tablet_manager.cc:1939] T 715274a681f447c594e8fcd38d6fd609 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.678032 754 tablet_service.cc:1467] Processing CreateTablet for tablet d243daba84ff4befa0c4e7bac5fe7476 (DEFAULT_TABLE table=default.Table_c06f4c91e37e4b6f95521174e05efb24 [id=9a6ed225ff3143f5bd2fc0f25324d322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.678463 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d243daba84ff4befa0c4e7bac5fe7476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.627702 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 715274a681f447c594e8fcd38d6fd609 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.680447 1165 log.cc:1198] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/379087c496d5401ebcbd1469ce1f68f3
I20250115 14:51:52.680630 741 tablet_service.cc:1467] Processing CreateTablet for tablet ab1a38186c5042ff851f57979ab80dbc (DEFAULT_TABLE table=default.Table_18aff4b27c7846fbaede10b093240587 [id=3e116aeb5bcb4e098de4e46730e3dd96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.680938 1165 ts_tablet_manager.cc:1939] T 379087c496d5401ebcbd1469ce1f68f3 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.636682 1162 tablet_bootstrap.cc:492] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.638943 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 745153c106844076bb020f9ed026c97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.643921 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet c56fe5b5b59b4017be3cb0ac877479b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.644277 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4259dfb8ee4b618c0bacc910addd0a with delete_type TABLET_DATA_DELETED (replaced by tablet 3a38a6bc32314396b497e7ae166a63f7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.661442 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e2495496b746abb568e78377c83d1e with delete_type TABLET_DATA_DELETED (replaced by tablet 4b90205ef15f470a99c013e7f2cf99c6) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.681334 1162 ts_tablet_manager.cc:1397] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.113s user 0.006s sys 0.000s
I20250115 14:51:52.661613 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet be18f1b6d8a24fabbe818811ef646c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.681695 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6c392518a464dab664e5f9321f1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.682225 1162 raft_consensus.cc:357] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.682433 1162 raft_consensus.cc:383] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.682509 1162 raft_consensus.cc:738] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.682735 1162 consensus_queue.cc:260] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.683261 1162 ts_tablet_manager.cc:1428] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:52.683562 1162 tablet_bootstrap.cc:492] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.684746 1165 tablet_replica.cc:331] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.684926 1165 raft_consensus.cc:2238] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.685076 1165 raft_consensus.cc:2267] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.685288 1169 raft_consensus.cc:491] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.685431 1169 raft_consensus.cc:513] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.686125 1169 leader_election.cc:290] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:52.687733 1165 ts_tablet_manager.cc:1905] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:52.688818 983 leader_election.cc:343] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 2b4259dfb8ee4b618c0bacc910addd0a
I20250115 14:51:52.689249 1167 tablet_replica.cc:331] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.689455 1167 raft_consensus.cc:2238] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.689679 1167 raft_consensus.cc:2267] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.690207 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab1a38186c5042ff851f57979ab80dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.691159 1167 ts_tablet_manager.cc:1905] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.694876 1162 tablet_bootstrap.cc:654] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.698104 889 tablet_service.cc:1467] Processing CreateTablet for tablet 4821224ac8f1472d8b1e8afc06a118a3 (DEFAULT_TABLE table=default.Table_bf7d10b07fa646a8a235ec0eeb725f64 [id=c5f831177088497dbf1fee130e231e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.698469 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4821224ac8f1472d8b1e8afc06a118a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.700949 879 tablet_service.cc:1467] Processing CreateTablet for tablet d243daba84ff4befa0c4e7bac5fe7476 (DEFAULT_TABLE table=default.Table_c06f4c91e37e4b6f95521174e05efb24 [id=9a6ed225ff3143f5bd2fc0f25324d322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.701328 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d243daba84ff4befa0c4e7bac5fe7476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.702723 1165 ts_tablet_manager.cc:1918] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.702857 1165 log.cc:1198] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/715274a681f447c594e8fcd38d6fd609
I20250115 14:51:52.703141 1165 ts_tablet_manager.cc:1939] T 715274a681f447c594e8fcd38d6fd609 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.704383 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]). Expected version 2 got 0
I20250115 14:51:52.705235 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ca27ab47d077491489383629eaaf1437" } modify_external_catalogs: true
I20250115 14:51:52.706126 885 tablet_service.cc:1467] Processing CreateTablet for tablet 745153c106844076bb020f9ed026c97a (DEFAULT_TABLE table=default.Table_031dcfe40eda4ff49338ab79a0cf9844 [id=9cda5b5b1bcf47dfbc5b5a8dcc69b44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.706502 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 745153c106844076bb020f9ed026c97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.706454 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3ec000089a47a2930dec382192cd6f (DEFAULT_TABLE table=default.Table_59fb977786f045519df5d218a7fd5e8d [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.706488 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ca27ab47d077491489383629eaaf1437" } modify_external_catalogs: true
I20250115 14:51:52.706827 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e3ec000089a47a2930dec382192cd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.706789 880 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7580b6245b41628874d9dcbc3366e2 (DEFAULT_TABLE table=default.Table_843fb099548148f29e7b622a4ad34b61 [id=085d5f79fc6c4bdb87752ecc55d89445]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.707051 884 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae60842a2d4486eb7651c3f710ee1e8 (DEFAULT_TABLE table=default.Table_edeb6add07054dcba5df71224ae8675c [id=ad886fc755c2448c93964215e2fd5903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.707288 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e7580b6245b41628874d9dcbc3366e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.707263 1207 raft_consensus.cc:491] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.707408 1207 raft_consensus.cc:513] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.707594 890 tablet_service.cc:1467] Processing CreateTablet for tablet f55c75e5febb407398b0bc06a1db9b1a (DEFAULT_TABLE table=default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.708066 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55c75e5febb407398b0bc06a1db9b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.708317 1207 leader_election.cc:290] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.708465 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c00f03edc74af8b084bb236487a347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.709165 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ae60842a2d4486eb7651c3f710ee1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.710680 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 379087c496d5401ebcbd1469ce1f68f3 (table default.Table_1d1bc2260e92451c99dfebfe09374e38 [id=959c2861880b40dcbb3780f3dff3b0f9]) successfully deleted
I20250115 14:51:52.711211 888 tablet_service.cc:1467] Processing CreateTablet for tablet ab1a38186c5042ff851f57979ab80dbc (DEFAULT_TABLE table=default.Table_18aff4b27c7846fbaede10b093240587 [id=3e116aeb5bcb4e098de4e46730e3dd96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.711539 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab1a38186c5042ff851f57979ab80dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.711658 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 715274a681f447c594e8fcd38d6fd609 (table default.Table_87a3775a6a0d41eabb46cb6f64e46c0d [id=d1df7ebdc5994a36800250155aefd024]) successfully deleted
I20250115 14:51:52.715117 1018 tablet_service.cc:1467] Processing CreateTablet for tablet d243daba84ff4befa0c4e7bac5fe7476 (DEFAULT_TABLE table=default.Table_c06f4c91e37e4b6f95521174e05efb24 [id=9a6ed225ff3143f5bd2fc0f25324d322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.715524 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 745153c106844076bb020f9ed026c97a (DEFAULT_TABLE table=default.Table_031dcfe40eda4ff49338ab79a0cf9844 [id=9cda5b5b1bcf47dfbc5b5a8dcc69b44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.715601 1022 tablet_service.cc:1467] Processing CreateTablet for tablet ab1a38186c5042ff851f57979ab80dbc (DEFAULT_TABLE table=default.Table_18aff4b27c7846fbaede10b093240587 [id=3e116aeb5bcb4e098de4e46730e3dd96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.716058 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae60842a2d4486eb7651c3f710ee1e8 (DEFAULT_TABLE table=default.Table_edeb6add07054dcba5df71224ae8675c [id=ad886fc755c2448c93964215e2fd5903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.716538 1023 tablet_service.cc:1467] Processing CreateTablet for tablet f55c75e5febb407398b0bc06a1db9b1a (DEFAULT_TABLE table=default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.716882 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3ec000089a47a2930dec382192cd6f (DEFAULT_TABLE table=default.Table_59fb977786f045519df5d218a7fd5e8d [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.717229 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 4b90205ef15f470a99c013e7f2cf99c6 (DEFAULT_TABLE table=default.Table_2e17c919cc6f4992b7d7b764ba964949 [id=017270e9a3ad41f18c34b3cfc2c4cd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.717602 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 4821224ac8f1472d8b1e8afc06a118a3 (DEFAULT_TABLE table=default.Table_bf7d10b07fa646a8a235ec0eeb725f64 [id=c5f831177088497dbf1fee130e231e37]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:52.722785 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b027233cfd314d0ebda472b4588d3f00 consensus state has no leader
I20250115 14:51:52.722755 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7580b6245b41628874d9dcbc3366e2 (DEFAULT_TABLE table=default.Table_843fb099548148f29e7b622a4ad34b61 [id=085d5f79fc6c4bdb87752ecc55d89445]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.723177 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 715274a681f447c594e8fcd38d6fd609 (table default.Table_87a3775a6a0d41eabb46cb6f64e46c0d [id=d1df7ebdc5994a36800250155aefd024]) successfully deleted
W20250115 14:51:52.723430 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 745153c106844076bb020f9ed026c97a because the tablet was not found. No further retry: Not found: Tablet not found: 745153c106844076bb020f9ed026c97a
W20250115 14:51:52.723635 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c56fe5b5b59b4017be3cb0ac877479b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c56fe5b5b59b4017be3cb0ac877479b4 already in progress: creating tablet
I20250115 14:51:52.724342 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 13c00f03edc74af8b084bb236487a347 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.724382 749 tablet_service.cc:1467] Processing CreateTablet for tablet 745153c106844076bb020f9ed026c97a (DEFAULT_TABLE table=default.Table_031dcfe40eda4ff49338ab79a0cf9844 [id=9cda5b5b1bcf47dfbc5b5a8dcc69b44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.724789 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 745153c106844076bb020f9ed026c97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.725962 746 tablet_service.cc:1467] Processing CreateTablet for tablet 4b90205ef15f470a99c013e7f2cf99c6 (DEFAULT_TABLE table=default.Table_2e17c919cc6f4992b7d7b764ba964949 [id=017270e9a3ad41f18c34b3cfc2c4cd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.726293 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b90205ef15f470a99c013e7f2cf99c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.726989 752 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7580b6245b41628874d9dcbc3366e2 (DEFAULT_TABLE table=default.Table_843fb099548148f29e7b622a4ad34b61 [id=085d5f79fc6c4bdb87752ecc55d89445]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.727274 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e7580b6245b41628874d9dcbc3366e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.727739 747 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae60842a2d4486eb7651c3f710ee1e8 (DEFAULT_TABLE table=default.Table_edeb6add07054dcba5df71224ae8675c [id=ad886fc755c2448c93964215e2fd5903]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.727645 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_910803aa49a042acba7e5a6a65b2e85d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.727933 751 tablet_service.cc:1467] Processing CreateTablet for tablet f55c75e5febb407398b0bc06a1db9b1a (DEFAULT_TABLE table=default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:52.728196 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_910803aa49a042acba7e5a6a65b2e85d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.728581 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b4259dfb8ee4b618c0bacc910addd0a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:52.728719 889 tablet_service.cc:1467] Processing CreateTablet for tablet 4b90205ef15f470a99c013e7f2cf99c6 (DEFAULT_TABLE table=default.Table_2e17c919cc6f4992b7d7b764ba964949 [id=017270e9a3ad41f18c34b3cfc2c4cd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.728946 756 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3ec000089a47a2930dec382192cd6f (DEFAULT_TABLE table=default.Table_59fb977786f045519df5d218a7fd5e8d [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.729123 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b90205ef15f470a99c013e7f2cf99c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.729542 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ae60842a2d4486eb7651c3f710ee1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.730811 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e3ec000089a47a2930dec382192cd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.731438 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55c75e5febb407398b0bc06a1db9b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.732321 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d837c64ad4503a224ced67ff1d053" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:52.734620 1167 ts_tablet_manager.cc:1918] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:52.734766 1167 log.cc:1198] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/379087c496d5401ebcbd1469ce1f68f3
W20250115 14:51:52.734714 848 leader_election.cc:343] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.735175 1167 ts_tablet_manager.cc:1939] T 379087c496d5401ebcbd1469ce1f68f3 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
W20250115 14:51:52.735296 984 leader_election.cc:343] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.735428 984 leader_election.cc:304] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:52.735848 1225 raft_consensus.cc:2743] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:52.736150 1167 tablet_replica.cc:331] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.736358 1167 raft_consensus.cc:2238] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.736515 1167 raft_consensus.cc:2267] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.737198 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c87fa8eb006845f5bf7b37b90f4c14c6" } new_table_name: "default.Table_30d9bde332c24d2a94267ffd2605a7cf" modify_external_catalogs: true
I20250115 14:51:52.738832 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_733144094e85412998e706e7a0f93dbf" } new_table_name: "default.Table_85673afa950047b1aee61fc1b7692be3" modify_external_catalogs: true
I20250115 14:51:52.740324 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 379087c496d5401ebcbd1469ce1f68f3 (table default.Table_1d1bc2260e92451c99dfebfe09374e38 [id=959c2861880b40dcbb3780f3dff3b0f9]) successfully deleted
I20250115 14:51:52.741214 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_edeb6add07054dcba5df71224ae8675c" } modify_external_catalogs: true
I20250115 14:51:52.741372 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_edeb6add07054dcba5df71224ae8675c" } modify_external_catalogs: true
I20250115 14:51:52.742678 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9ddc7a7e450f465bb0113fdb6ea442ed" } new_table_name: "default.Table_48222c861a6e490a82607bb753a21cf6" modify_external_catalogs: true
I20250115 14:51:52.749749 1207 raft_consensus.cc:491] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.749950 1207 raft_consensus.cc:513] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.750649 1207 leader_election.cc:290] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.751268 1225 raft_consensus.cc:491] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.751427 1225 raft_consensus.cc:513] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.752113 1225 leader_election.cc:290] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:52.752760 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c368b4ef34f147b6984aee76adde3cf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.753145 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c6c392518a464dab664e5f9321f1f6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:51:52.753293 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c368b4ef34f147b6984aee76adde3cf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.753377 912 raft_consensus.cc:2463] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:52.753808 983 leader_election.cc:304] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:52.754144 1225 raft_consensus.cc:2798] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:52.754242 1225 raft_consensus.cc:491] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:52.754315 1225 raft_consensus.cc:3054] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:52.758198 1225 raft_consensus.cc:513] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.759336 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c6c392518a464dab664e5f9321f1f6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:52.759589 912 raft_consensus.cc:3054] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:52.759904 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 74bfda3861d148bc8b67ea071909d22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.761566 1162 tablet_bootstrap.cc:492] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.761740 1162 ts_tablet_manager.cc:1397] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.000s
I20250115 14:51:52.762537 1162 raft_consensus.cc:357] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.762761 1162 raft_consensus.cc:383] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.762823 1162 raft_consensus.cc:738] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.763046 1162 consensus_queue.cc:260] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.763543 1162 ts_tablet_manager.cc:1428] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:52.764788 912 raft_consensus.cc:2463] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:52.765887 983 leader_election.cc:304] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:52.766189 1169 raft_consensus.cc:2798] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:52.766845 1167 ts_tablet_manager.cc:1905] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.766888 1169 raft_consensus.cc:695] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:52.767194 1169 consensus_queue.cc:237] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.768174 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d837c64ad4503a224ced67ff1d053" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:52.768275 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c00f03edc74af8b084bb236487a347" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:52.768411 1165 tablet_replica.cc:331] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.768435 1049 raft_consensus.cc:2463] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:52.768615 1165 raft_consensus.cc:2238] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.768797 1165 raft_consensus.cc:2267] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.769039 847 leader_election.cc:304] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:52.769482 847 leader_election.cc:343] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.769717 1207 raft_consensus.cc:2798] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:52.769810 1207 raft_consensus.cc:491] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:52.769917 1207 raft_consensus.cc:3054] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:52.771513 1168 tablet_bootstrap.cc:492] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.771634 1168 ts_tablet_manager.cc:1397] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.099s user 0.029s sys 0.022s
I20250115 14:51:52.772759 1207 raft_consensus.cc:513] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.773255 1168 raft_consensus.cc:357] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.773504 1168 raft_consensus.cc:383] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.773595 1168 raft_consensus.cc:738] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.773834 1168 consensus_queue.cc:260] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.775247 1168 ts_tablet_manager.cc:1428] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:51:52.775514 1168 tablet_bootstrap.cc:492] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.778934 1167 ts_tablet_manager.cc:1918] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.779083 1167 log.cc:1198] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/be18f1b6d8a24fabbe818811ef646c62
I20250115 14:51:52.779458 1167 ts_tablet_manager.cc:1939] T be18f1b6d8a24fabbe818811ef646c62 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.779824 1168 tablet_bootstrap.cc:654] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.779908 1167 tablet_replica.cc:331] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.780107 1167 raft_consensus.cc:2238] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.780303 1167 raft_consensus.cc:2267] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.780797 1167 ts_tablet_manager.cc:1905] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.782047 1225 leader_election.cc:290] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:52.783259 1207 leader_election.cc:290] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.783812 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d837c64ad4503a224ced67ff1d053" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:52.784024 1049 raft_consensus.cc:3054] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:52.784845 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d243daba84ff4befa0c4e7bac5fe7476. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.786805 1049 raft_consensus.cc:2463] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:52.786911 1167 ts_tablet_manager.cc:1918] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.787020 1167 log.cc:1198] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/66c6c392518a464dab664e5f9321f1f6
I20250115 14:51:52.787230 847 leader_election.cc:304] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:52.787427 1167 ts_tablet_manager.cc:1939] T 66c6c392518a464dab664e5f9321f1f6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.787684 1207 raft_consensus.cc:2798] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:52.788149 1167 tablet_replica.cc:331] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.788379 1207 raft_consensus.cc:695] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:52.788458 1167 raft_consensus.cc:2238] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.788569 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 745153c106844076bb020f9ed026c97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.788614 1167 raft_consensus.cc:2267] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.788687 1207 consensus_queue.cc:237] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.789202 1168 tablet_bootstrap.cc:492] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.789727 1168 ts_tablet_manager.cc:1397] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20250115 14:51:52.790441 1168 raft_consensus.cc:357] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.790630 1168 raft_consensus.cc:383] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.790706 1168 raft_consensus.cc:738] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.790910 1168 consensus_queue.cc:260] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.791357 1167 ts_tablet_manager.cc:1905] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.791376 1168 ts_tablet_manager.cc:1428] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:52.791697 1168 tablet_bootstrap.cc:492] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.792084 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab1a38186c5042ff851f57979ab80dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.793009 1162 tablet_bootstrap.cc:492] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.793309 1168 tablet_bootstrap.cc:654] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.794605 1162 tablet_bootstrap.cc:654] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.796066 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4821224ac8f1472d8b1e8afc06a118a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.796435 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b90205ef15f470a99c013e7f2cf99c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.799974 1162 tablet_bootstrap.cc:492] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.800104 1162 ts_tablet_manager.cc:1397] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20250115 14:51:52.800150 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e3ec000089a47a2930dec382192cd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.800870 1162 raft_consensus.cc:357] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.801061 1162 raft_consensus.cc:383] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.801158 1162 raft_consensus.cc:738] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.801376 1162 consensus_queue.cc:260] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.801879 1162 ts_tablet_manager.cc:1428] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:52.802147 1162 tablet_bootstrap.cc:492] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.805219 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f55c75e5febb407398b0bc06a1db9b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.806816 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ae60842a2d4486eb7651c3f710ee1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.814955 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f224f857b1644596853acd9e3a2a4a5f
I20250115 14:51:52.815421 454 catalog_manager.cc:6596] ReplaceTablet: tablet f224f857b1644596853acd9e3a2a4a5f deleted and replaced by tablet e50462626f914fb1b0f440c48f30e72d
W20250115 14:51:52.815428 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet bd6d837c64ad4503a224ced67ff1d053 consensus state has no leader
I20250115 14:51:52.815856 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet be18f1b6d8a24fabbe818811ef646c62 (table default.Table_40664b92268d4ba39f3f5852835f97d2 [id=da39f1615db74469824c3bf8eb74d260]) successfully deleted
I20250115 14:51:52.816310 1168 tablet_bootstrap.cc:492] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.816464 1168 ts_tablet_manager.cc:1397] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.002s
I20250115 14:51:52.816725 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 66c6c392518a464dab664e5f9321f1f6 (table default.Table_7281d9e98ae843dc915b4932d3d7d50b [id=52b6d8da69f74139a394cafe66e48a35]) successfully deleted
I20250115 14:51:52.817258 1168 raft_consensus.cc:357] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.817461 1168 raft_consensus.cc:383] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.817543 1168 raft_consensus.cc:738] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.817767 1168 consensus_queue.cc:260] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.818269 1168 ts_tablet_manager.cc:1428] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:52.818559 1168 tablet_bootstrap.cc:492] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:51:52.821202 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 745153c106844076bb020f9ed026c97a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 745153c106844076bb020f9ed026c97a already in progress: creating tablet
W20250115 14:51:52.822891 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c56fe5b5b59b4017be3cb0ac877479b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c56fe5b5b59b4017be3cb0ac877479b4 already in progress: creating tablet
I20250115 14:51:52.824265 1162 tablet_bootstrap.cc:654] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.824537 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f224f857b1644596853acd9e3a2a4a5f with delete_type TABLET_DATA_DELETED (replaced by tablet e50462626f914fb1b0f440c48f30e72d) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:52.825604 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f224f857b1644596853acd9e3a2a4a5f with delete_type TABLET_DATA_DELETED (replaced by tablet e50462626f914fb1b0f440c48f30e72d) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.825984 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c6c392518a464dab664e5f9321f1f6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:52.826982 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c6c392518a464dab664e5f9321f1f6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:52.834384 742 tablet_service.cc:1514] Processing DeleteTablet for tablet f224f857b1644596853acd9e3a2a4a5f with delete_type TABLET_DATA_DELETED (replaced by tablet e50462626f914fb1b0f440c48f30e72d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.834668 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd6d837c64ad4503a224ced67ff1d053" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:52.834982 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13c00f03edc74af8b084bb236487a347" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:52.835120 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f224f857b1644596853acd9e3a2a4a5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f224f857b1644596853acd9e3a2a4a5f already in progress: creating tablet
W20250115 14:51:52.835120 848 leader_election.cc:343] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:52.836342 848 leader_election.cc:343] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.836460 848 leader_election.cc:304] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:52.836865 1174 raft_consensus.cc:2743] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:52.843031 1174 raft_consensus.cc:491] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.843180 1174 raft_consensus.cc:513] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.843878 1174 leader_election.cc:290] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.845018 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6326b1526191464485855b657e34a2b9" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:52.845492 848 leader_election.cc:343] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.851686 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e7580b6245b41628874d9dcbc3366e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:52.856374 984 leader_election.cc:343] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.856545 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6326b1526191464485855b657e34a2b9" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:52.856830 984 leader_election.cc:343] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.857228 1162 tablet_bootstrap.cc:492] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
W20250115 14:51:52.857259 847 leader_election.cc:343] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.857370 1162 ts_tablet_manager.cc:1397] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.002s
I20250115 14:51:52.857383 847 leader_election.cc:304] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:52.857792 1174 raft_consensus.cc:2743] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:52.858116 1162 raft_consensus.cc:357] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.858307 1162 raft_consensus.cc:383] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.858372 1162 raft_consensus.cc:738] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.858589 1162 consensus_queue.cc:260] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.859076 1162 ts_tablet_manager.cc:1428] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:52.859433 1162 tablet_bootstrap.cc:492] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.861431 1162 tablet_bootstrap.cc:654] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.863054 1168 tablet_bootstrap.cc:654] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.864596 1162 tablet_bootstrap.cc:492] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.864741 1162 ts_tablet_manager.cc:1397] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250115 14:51:52.865458 1162 raft_consensus.cc:357] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.865665 1162 raft_consensus.cc:383] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.865759 1162 raft_consensus.cc:738] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.865971 1162 consensus_queue.cc:260] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.866479 1162 ts_tablet_manager.cc:1428] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:52.866748 1162 tablet_bootstrap.cc:492] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.866878 1168 tablet_bootstrap.cc:492] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.867031 1168 ts_tablet_manager.cc:1397] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.001s
I20250115 14:51:52.867803 1168 raft_consensus.cc:357] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.868034 1168 raft_consensus.cc:383] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.868106 1168 raft_consensus.cc:738] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.868461 1168 consensus_queue.cc:260] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.869079 1168 ts_tablet_manager.cc:1428] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:52.869417 1162 tablet_bootstrap.cc:654] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.869812 1168 tablet_bootstrap.cc:492] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.871784 1168 tablet_bootstrap.cc:654] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.877465 1164 tablet_bootstrap.cc:492] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.878201 1162 tablet_bootstrap.cc:492] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.878374 1162 ts_tablet_manager.cc:1397] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20250115 14:51:52.879132 1164 ts_tablet_manager.cc:1397] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.440s user 0.039s sys 0.037s
I20250115 14:51:52.879146 1162 raft_consensus.cc:357] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.879369 1162 raft_consensus.cc:383] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.879433 1162 raft_consensus.cc:738] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.879695 1162 consensus_queue.cc:260] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.880182 1162 ts_tablet_manager.cc:1428] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:52.880494 1162 tablet_bootstrap.cc:492] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.882140 1162 tablet_bootstrap.cc:654] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.882337 1167 ts_tablet_manager.cc:1918] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.882511 1167 log.cc:1198] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/c322301320ba425a94d7321c08f7f9e6
I20250115 14:51:52.882885 1167 ts_tablet_manager.cc:1939] T c322301320ba425a94d7321c08f7f9e6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.883328 1164 raft_consensus.cc:357] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.884282 1165 ts_tablet_manager.cc:1905] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.884375 1164 raft_consensus.cc:383] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.884827 1164 raft_consensus.cc:738] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.885506 1164 consensus_queue.cc:260] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.890714 1164 ts_tablet_manager.cc:1428] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250115 14:51:52.892858 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet e115bdfa1d69421890122f9afdc23b29 with delete_type TABLET_DATA_DELETED (replaced by tablet f312f1e3a33a404ca7284c713859f156) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.892920 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet c322301320ba425a94d7321c08f7f9e6 (table default.Table_dd152d863f394374a58bf845d3d259e2 [id=d5216eacff2b46ad96642b0ef7fd6403]) successfully deleted
I20250115 14:51:52.894898 1168 tablet_bootstrap.cc:492] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.895067 1168 ts_tablet_manager.cc:1397] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.003s
I20250115 14:51:52.895833 1168 raft_consensus.cc:357] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.896030 1168 raft_consensus.cc:383] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.896111 1168 raft_consensus.cc:738] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.896324 1168 consensus_queue.cc:260] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.896809 1168 ts_tablet_manager.cc:1428] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:52.897073 1168 tablet_bootstrap.cc:492] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.898213 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 58cca6072e4e4f608c6a45be279304f8
I20250115 14:51:52.898819 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 3fd72a0d586d4a2c817f6a39daa07063 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.899030 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cca6072e4e4f608c6a45be279304f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.899343 1168 tablet_bootstrap.cc:654] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.899395 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cca6072e4e4f608c6a45be279304f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:52.900220 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4821224ac8f1472d8b1e8afc06a118a3 consensus state has no leader
I20250115 14:51:52.900305 1167 tablet_replica.cc:331] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.900496 1167 raft_consensus.cc:2238] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.900521 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 74bfda3861d148bc8b67ea071909d22d from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.900672 1167 raft_consensus.cc:2267] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.903863 1167 ts_tablet_manager.cc:1905] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.907650 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cca6072e4e4f608c6a45be279304f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:52.908896 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 58cca6072e4e4f608c6a45be279304f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58cca6072e4e4f608c6a45be279304f8 already in progress: creating tablet
I20250115 14:51:52.916471 1165 ts_tablet_manager.cc:1918] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.916617 1165 log.cc:1198] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/2b4259dfb8ee4b618c0bacc910addd0a
I20250115 14:51:52.916970 1165 ts_tablet_manager.cc:1939] T 2b4259dfb8ee4b618c0bacc910addd0a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.917977 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 2b4259dfb8ee4b618c0bacc910addd0a (table default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]) successfully deleted
I20250115 14:51:52.918337 1165 tablet_replica.cc:331] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.918548 1165 raft_consensus.cc:2238] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.918700 1165 raft_consensus.cc:2267] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.920477 1165 ts_tablet_manager.cc:1905] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:52.921772 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea0f104c1a71404591072e7a36b276b0 consensus state has no leader
W20250115 14:51:52.924257 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb792ca1b3f54def92f504ec8682d2c7 consensus state has no leader
I20250115 14:51:52.924546 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 3ae60842a2d4486eb7651c3f710ee1e8 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.925590 1167 ts_tablet_manager.cc:1918] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.925709 1167 log.cc:1198] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/13c00f03edc74af8b084bb236487a347
I20250115 14:51:52.926066 1167 ts_tablet_manager.cc:1939] T 13c00f03edc74af8b084bb236487a347 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:52.926620 1174 raft_consensus.cc:491] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.926637 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3ae60842a2d4486eb7651c3f710ee1e8
I20250115 14:51:52.926752 1174 raft_consensus.cc:513] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.927448 1174 leader_election.cc:290] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.927817 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ae60842a2d4486eb7651c3f710ee1e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:52.928937 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22edaa6e70d942ba98777973476f449d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:52.929041 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ae60842a2d4486eb7651c3f710ee1e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.929700 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ae60842a2d4486eb7651c3f710ee1e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:52.930145 847 leader_election.cc:343] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:52.932567 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d77ff794a83348c29391786666987db1 consensus state has no leader
I20250115 14:51:52.932842 1165 ts_tablet_manager.cc:1918] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.932983 1165 log.cc:1198] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/70e2495496b746abb568e78377c83d1e
I20250115 14:51:52.933019 1174 raft_consensus.cc:491] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.933194 1174 raft_consensus.cc:513] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.933413 1165 ts_tablet_manager.cc:1939] T 70e2495496b746abb568e78377c83d1e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:52.933848 1174 leader_election.cc:290] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.933800 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_49eb379e719f4677b13269d15ef7efad" } modify_external_catalogs: true
I20250115 14:51:52.934008 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_49eb379e719f4677b13269d15ef7efad" } modify_external_catalogs: true
I20250115 14:51:52.934727 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9126b20cb8474e8e9817459ddf0b9b0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.934846 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d51e53b4ffa45bfbb9553d7a5870844" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:52.935204 1165 tablet_replica.cc:331] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:52.935374 1165 raft_consensus.cc:2238] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:51:52.935359 847 leader_election.cc:343] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.935554 1165 raft_consensus.cc:2267] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.935874 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c0256da787104a42a1bc1e3946e11fd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.936355 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22edaa6e70d942ba98777973476f449d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:52.936538 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_91244a7b0fa14291bcc2e9a2d776b4ae" } modify_external_catalogs: true
I20250115 14:51:52.936722 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_91244a7b0fa14291bcc2e9a2d776b4ae" } modify_external_catalogs: true
I20250115 14:51:52.936920 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b876bcff1252445ca4b6881550c9dd82" } modify_external_catalogs: true
I20250115 14:51:52.937085 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b876bcff1252445ca4b6881550c9dd82" } modify_external_catalogs: true
I20250115 14:51:52.937222 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d51e53b4ffa45bfbb9553d7a5870844" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:52.937680 848 leader_election.cc:343] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.937808 848 leader_election.cc:304] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:52.938127 1207 raft_consensus.cc:2743] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:52.938192 848 leader_election.cc:343] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.938347 848 leader_election.cc:304] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:52.938866 1174 raft_consensus.cc:2743] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:52.938783 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d7378da7b662410d864c634dd5e34926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:52.939292 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9126b20cb8474e8e9817459ddf0b9b0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.939402 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 70e2495496b746abb568e78377c83d1e (table default.Table_2e17c919cc6f4992b7d7b764ba964949 [id=017270e9a3ad41f18c34b3cfc2c4cd42]) successfully deleted
W20250115 14:51:52.940032 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7378da7b662410d864c634dd5e34926 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.937238 1164 tablet_bootstrap.cc:492] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:52.941494 1168 tablet_bootstrap.cc:492] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.941640 1168 ts_tablet_manager.cc:1397] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.013s
W20250115 14:51:52.940310 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3ae60842a2d4486eb7651c3f710ee1e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ae60842a2d4486eb7651c3f710ee1e8 already in progress: creating tablet
W20250115 14:51:52.943017 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0256da787104a42a1bc1e3946e11fd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.942935 1168 raft_consensus.cc:357] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.943161 1168 raft_consensus.cc:383] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.943226 1168 raft_consensus.cc:738] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.943467 1168 consensus_queue.cc:260] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.944106 1168 ts_tablet_manager.cc:1428] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:52.944418 1168 tablet_bootstrap.cc:492] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.945003 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e2495496b746abb568e78377c83d1e with delete_type TABLET_DATA_DELETED (replaced by tablet 4b90205ef15f470a99c013e7f2cf99c6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.946034 1166 tablet_replica.cc:331] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:52.946053 1168 tablet_bootstrap.cc:654] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.946215 1166 raft_consensus.cc:2238] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.946378 1166 raft_consensus.cc:2267] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.946511 1207 raft_consensus.cc:491] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:52.946645 1207 raft_consensus.cc:513] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.946856 1164 tablet_bootstrap.cc:654] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.948127 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d43cd66b18f4649894a629637037822" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:52.948501 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d43cd66b18f4649894a629637037822" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:52.948760 847 leader_election.cc:343] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.948928 1207 leader_election.cc:290] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:51:52.950307 848 leader_election.cc:343] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:52.950449 848 leader_election.cc:304] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:52.950690 1162 tablet_bootstrap.cc:492] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.950848 1162 ts_tablet_manager.cc:1397] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.070s user 0.014s sys 0.015s
I20250115 14:51:52.950946 1207 raft_consensus.cc:2743] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:52.951789 1162 raft_consensus.cc:357] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.951992 1162 raft_consensus.cc:383] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.952076 1162 raft_consensus.cc:738] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:52.952355 1162 consensus_queue.cc:260] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.952876 1162 ts_tablet_manager.cc:1428] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:52.953564 1162 tablet_bootstrap.cc:492] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.954102 1165 ts_tablet_manager.cc:1905] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.956152 1166 ts_tablet_manager.cc:1905] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:52.959661 1162 tablet_bootstrap.cc:654] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.963037 1166 ts_tablet_manager.cc:1918] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:52.963222 1166 log.cc:1198] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/70e2495496b746abb568e78377c83d1e
I20250115 14:51:52.963570 1166 ts_tablet_manager.cc:1939] T 70e2495496b746abb568e78377c83d1e P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:52.966861 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 70e2495496b746abb568e78377c83d1e (table default.Table_2e17c919cc6f4992b7d7b764ba964949 [id=017270e9a3ad41f18c34b3cfc2c4cd42]) successfully deleted
I20250115 14:51:52.968582 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_de5e692c0fa544a5975b1cc474c6a6d5" } new_table_name: "default.Table_56a365aae805419683d1f4dfd97951cc" modify_external_catalogs: true
I20250115 14:51:52.968786 1168 tablet_bootstrap.cc:492] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:52.968943 1168 ts_tablet_manager.cc:1397] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.005s
I20250115 14:51:52.969542 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.969767 1168 raft_consensus.cc:357] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.969969 1168 raft_consensus.cc:383] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.969063 1164 tablet_bootstrap.cc:492] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:52.969569 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 70e2495496b746abb568e78377c83d1e with delete_type TABLET_DATA_DELETED (replaced by tablet 4b90205ef15f470a99c013e7f2cf99c6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:52.970050 1168 raft_consensus.cc:738] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
W20250115 14:51:52.970125 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.970300 1168 consensus_queue.cc:260] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:52.970577 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 70e2495496b746abb568e78377c83d1e because the tablet was not found. No further retry: Not found: Tablet not found: 70e2495496b746abb568e78377c83d1e
I20250115 14:51:52.972579 1168 ts_tablet_manager.cc:1428] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250115 14:51:52.972864 1168 tablet_bootstrap.cc:492] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:52.974895 1168 tablet_bootstrap.cc:654] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.975807 1162 tablet_bootstrap.cc:492] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:52.975986 1162 ts_tablet_manager.cc:1397] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.001s
I20250115 14:51:52.977980 1162 raft_consensus.cc:357] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.978199 1162 raft_consensus.cc:383] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:52.978080 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0dbb36cfea4d45d88c5ef963c153cf77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:52.978274 1162 raft_consensus.cc:738] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
W20250115 14:51:52.978613 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dbb36cfea4d45d88c5ef963c153cf77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:52.978540 1162 consensus_queue.cc:260] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:52.979036 1162 ts_tablet_manager.cc:1428] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:52.979763 1162 tablet_bootstrap.cc:492] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:52.981606 1162 tablet_bootstrap.cc:654] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:52.981601 753 tablet_service.cc:1467] Processing CreateTablet for tablet 259ac764ca844df9b6c0ce1688a188da (DEFAULT_TABLE table=default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.982014 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259ac764ca844df9b6c0ce1688a188da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.982882 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d6a9fb932c5b4545a4609d568f0e5494 (table default.Table_8f6e7d6b890e4a449f170811c030b6f5 [id=3f3fecceceb3464bb12462bb3041b458]). Expected version 2 got 0
W20250115 14:51:52.984575 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f960c88df14242c5ada1ecf9d8e68196 consensus state has no leader
I20250115 14:51:52.984820 452 master_service.cc:946] ReplaceTablet: received request to replace tablet ea0f104c1a71404591072e7a36b276b0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:52.986757 1167 tablet_replica.cc:331] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:52.986971 1167 raft_consensus.cc:2238] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:52.987133 1167 raft_consensus.cc:2267] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:52.987738 754 tablet_service.cc:1467] Processing CreateTablet for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 (DEFAULT_TABLE table=default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.988102 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4fb2d33f3c745beb4a4ba036e23a8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.989358 884 tablet_service.cc:1467] Processing CreateTablet for tablet 4239747374aa4d6d866961430786677c (DEFAULT_TABLE table=default.Table_1653219949dc4fdeb257d0ba8c88c231 [id=8941c6c27037475990cf318614854a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.989718 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4239747374aa4d6d866961430786677c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.989667 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 259ac764ca844df9b6c0ce1688a188da (DEFAULT_TABLE table=default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.989785 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 13c00f03edc74af8b084bb236487a347 (table default.Table_64abfdac91a54d6bb59ccfc7791a170c [id=358f8fad6d944439847ea4ce82f1213a]) successfully deleted
I20250115 14:51:52.990032 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259ac764ca844df9b6c0ce1688a188da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.992604 747 tablet_service.cc:1467] Processing CreateTablet for tablet 4239747374aa4d6d866961430786677c (DEFAULT_TABLE table=default.Table_1653219949dc4fdeb257d0ba8c88c231 [id=8941c6c27037475990cf318614854a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.992586 885 tablet_service.cc:1467] Processing CreateTablet for tablet 5027fba8fed34db59b484918256abc45 (DEFAULT_TABLE table=default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.992633 1022 tablet_service.cc:1467] Processing CreateTablet for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 (DEFAULT_TABLE table=default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.992950 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5027fba8fed34db59b484918256abc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.992961 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4239747374aa4d6d866961430786677c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.993001 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4fb2d33f3c745beb4a4ba036e23a8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.996383 890 tablet_service.cc:1467] Processing CreateTablet for tablet 86f0144904ae4008945ccf325039dac1 (DEFAULT_TABLE table=default.Table_6674d45a98dc47b4b3c4abef46e52872 [id=ab6f49066e884652b77a0b9a791575c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.996546 751 tablet_service.cc:1467] Processing CreateTablet for tablet 5027fba8fed34db59b484918256abc45 (DEFAULT_TABLE table=default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.996579 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 4239747374aa4d6d866961430786677c (DEFAULT_TABLE table=default.Table_1653219949dc4fdeb257d0ba8c88c231 [id=8941c6c27037475990cf318614854a27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:52.996752 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f0144904ae4008945ccf325039dac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.996858 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5027fba8fed34db59b484918256abc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:52.996958 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4239747374aa4d6d866961430786677c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.000599 752 tablet_service.cc:1467] Processing CreateTablet for tablet 86f0144904ae4008945ccf325039dac1 (DEFAULT_TABLE table=default.Table_6674d45a98dc47b4b3c4abef46e52872 [id=ab6f49066e884652b77a0b9a791575c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.000614 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 5027fba8fed34db59b484918256abc45 (DEFAULT_TABLE table=default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.000788 879 tablet_service.cc:1467] Processing CreateTablet for tablet 36685a04e7454f6592f6ab83c356f466 (DEFAULT_TABLE table=default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.000931 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f0144904ae4008945ccf325039dac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.000954 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5027fba8fed34db59b484918256abc45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.001137 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36685a04e7454f6592f6ab83c356f466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.004457 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 86f0144904ae4008945ccf325039dac1 (DEFAULT_TABLE table=default.Table_6674d45a98dc47b4b3c4abef46e52872 [id=ab6f49066e884652b77a0b9a791575c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.004575 749 tablet_service.cc:1467] Processing CreateTablet for tablet 36685a04e7454f6592f6ab83c356f466 (DEFAULT_TABLE table=default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.004827 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f0144904ae4008945ccf325039dac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.004876 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36685a04e7454f6592f6ab83c356f466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.008549 741 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcfab1e1f0349739eff636fb1b04806 (DEFAULT_TABLE table=default.Table_fa872457db7140b6a13bfc998e32581b [id=4e1c43b0e14844babf0bb0cf8032a3fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.008630 889 tablet_service.cc:1467] Processing CreateTablet for tablet 259ac764ca844df9b6c0ce1688a188da (DEFAULT_TABLE table=default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.008919 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bcfab1e1f0349739eff636fb1b04806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.009074 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259ac764ca844df9b6c0ce1688a188da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.011674 891 tablet_service.cc:1467] Processing CreateTablet for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 (DEFAULT_TABLE table=default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.012090 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4fb2d33f3c745beb4a4ba036e23a8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.014873 746 tablet_service.cc:1467] Processing CreateTablet for tablet 07a63a0a8bb44fe9be0740fc777e610c (DEFAULT_TABLE table=default.Table_2a2f7c950304406484083d56f0b3177e [id=685971a2443b4b16a9af9f3a57d132c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.015228 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a63a0a8bb44fe9be0740fc777e610c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.008229 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 36685a04e7454f6592f6ab83c356f466 (DEFAULT_TABLE table=default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.016491 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36685a04e7454f6592f6ab83c356f466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.018586 1165 ts_tablet_manager.cc:1918] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.018723 1165 log.cc:1198] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/be18f1b6d8a24fabbe818811ef646c62
I20250115 14:51:53.019119 1165 ts_tablet_manager.cc:1939] T be18f1b6d8a24fabbe818811ef646c62 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.021049 1167 ts_tablet_manager.cc:1905] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.022905 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcfab1e1f0349739eff636fb1b04806 (DEFAULT_TABLE table=default.Table_fa872457db7140b6a13bfc998e32581b [id=4e1c43b0e14844babf0bb0cf8032a3fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.023339 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bcfab1e1f0349739eff636fb1b04806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.024588 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 07a63a0a8bb44fe9be0740fc777e610c (DEFAULT_TABLE table=default.Table_2a2f7c950304406484083d56f0b3177e [id=685971a2443b4b16a9af9f3a57d132c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.024984 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a63a0a8bb44fe9be0740fc777e610c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.029072 890 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcfab1e1f0349739eff636fb1b04806 (DEFAULT_TABLE table=default.Table_fa872457db7140b6a13bfc998e32581b [id=4e1c43b0e14844babf0bb0cf8032a3fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.029937 879 tablet_service.cc:1467] Processing CreateTablet for tablet 07a63a0a8bb44fe9be0740fc777e610c (DEFAULT_TABLE table=default.Table_2a2f7c950304406484083d56f0b3177e [id=685971a2443b4b16a9af9f3a57d132c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.030397 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bcfab1e1f0349739eff636fb1b04806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.036435 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a63a0a8bb44fe9be0740fc777e610c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.043083 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_490c22b7496a4d7db6bc6955e246363b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.043716 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_490c22b7496a4d7db6bc6955e246363b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.044667 1207 raft_consensus.cc:491] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.044826 1207 raft_consensus.cc:513] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.045439 1207 leader_election.cc:290] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:52.978091 1164 ts_tablet_manager.cc:1397] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.006s
I20250115 14:51:53.046088 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5bc6131944fcab9c550caac4f8d85" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:53.046303 1049 raft_consensus.cc:2463] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:53.046366 1164 raft_consensus.cc:357] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.046579 1164 raft_consensus.cc:383] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.046639 1164 raft_consensus.cc:738] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.046726 847 leader_election.cc:304] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:53.046869 1164 consensus_queue.cc:260] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.047293 1207 raft_consensus.cc:2798] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:53.047375 1207 raft_consensus.cc:491] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:53.047466 1207 raft_consensus.cc:3054] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.049024 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d809e208790b4c1a911708b393be6a3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.049583 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d809e208790b4c1a911708b393be6a3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.050483 1207 raft_consensus.cc:513] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.051632 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5bc6131944fcab9c550caac4f8d85" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:53.051878 1049 raft_consensus.cc:3054] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.054363 1167 ts_tablet_manager.cc:1918] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.054528 1167 log.cc:1198] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/74bfda3861d148bc8b67ea071909d22d
I20250115 14:51:53.054737 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5bc6131944fcab9c550caac4f8d85" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.054920 1167 ts_tablet_manager.cc:1939] T 74bfda3861d148bc8b67ea071909d22d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.054744 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5bc6131944fcab9c550caac4f8d85" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:53.055444 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 74bfda3861d148bc8b67ea071909d22d (table default.Table_a78c10f9e6384d7d89d3add6f1371429 [id=2c59ffcdee944c4289ad5dd38dc2e8cc]) successfully deleted
W20250115 14:51:53.057431 848 leader_election.cc:343] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:53.057801 848 leader_election.cc:343] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.058699 1164 ts_tablet_manager.cc:1428] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250115 14:51:53.052227 1207 leader_election.cc:290] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.060163 448 master_service.cc:946] ReplaceTablet: received request to replace tablet a6b76cb343fb4456ac07fd3c6e9888de from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.060544 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bbd41d946d245b78e6686f8fae55449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.061098 1166 tablet_replica.cc:331] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:53.061250 1166 raft_consensus.cc:2238] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.061398 1166 raft_consensus.cc:2267] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.063361 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3fd72a0d586d4a2c817f6a39daa07063
I20250115 14:51:53.063925 455 catalog_manager.cc:6596] ReplaceTablet: tablet 3fd72a0d586d4a2c817f6a39daa07063 deleted and replaced by tablet d7309ecb11d844848638c042c9bbe5f1
I20250115 14:51:53.064260 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd72a0d586d4a2c817f6a39daa07063 with delete_type TABLET_DATA_DELETED (replaced by tablet d7309ecb11d844848638c042c9bbe5f1) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.064429 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd72a0d586d4a2c817f6a39daa07063 with delete_type TABLET_DATA_DELETED (replaced by tablet d7309ecb11d844848638c042c9bbe5f1) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.064585 1167 tablet_replica.cc:331] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.064783 1167 raft_consensus.cc:2238] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.064971 1167 raft_consensus.cc:2267] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.065755 1167 ts_tablet_manager.cc:1905] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.067432 1166 ts_tablet_manager.cc:1905] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:53.051232 1221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 14:51:53.068728 1221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 14:51:53.068868 1221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250115 14:51:53.073793 1049 raft_consensus.cc:2463] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:53.074466 847 leader_election.cc:304] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:53.076014 441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20250115 14:51:53.077214 441 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b -> Peer 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227): Couldn't send request to peer 1cc2e23c560f4096be7b14006bdfc525. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250115 14:51:53.077493 1167 ts_tablet_manager.cc:1918] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.077620 1167 log.cc:1198] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f224f857b1644596853acd9e3a2a4a5f
I20250115 14:51:53.077983 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 24751e606ad94675b650d67a3738481e
I20250115 14:51:53.078014 1167 ts_tablet_manager.cc:1939] T f224f857b1644596853acd9e3a2a4a5f P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.078665 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 24751e606ad94675b650d67a3738481e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.078716 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a6b76cb343fb4456ac07fd3c6e9888de
I20250115 14:51:53.079603 753 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b76cb343fb4456ac07fd3c6e9888de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.079793 1209 raft_consensus.cc:2798] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:53.079941 1209 raft_consensus.cc:695] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:53.080228 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f224f857b1644596853acd9e3a2a4a5f (table default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]) successfully deleted
I20250115 14:51:53.080236 1209 consensus_queue.cc:237] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.080981 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 24751e606ad94675b650d67a3738481e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.081377 890 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b76cb343fb4456ac07fd3c6e9888de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.083181 1166 ts_tablet_manager.cc:1918] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.083316 1166 log.cc:1198] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9bbd41d946d245b78e6686f8fae55449
I20250115 14:51:53.083659 1166 ts_tablet_manager.cc:1939] T 9bbd41d946d245b78e6686f8fae55449 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:53.084147 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9bbd41d946d245b78e6686f8fae55449 (table default.Table_c128b7a02c834993a7068d76854faa7d [id=4745e3d5a5ba4176ad2b4f72b09846c5]) successfully deleted
W20250115 14:51:53.084679 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3fd72a0d586d4a2c817f6a39daa07063 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd72a0d586d4a2c817f6a39daa07063 already in progress: creating tablet
W20250115 14:51:53.084940 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 24751e606ad94675b650d67a3738481e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24751e606ad94675b650d67a3738481e already in progress: creating tablet
W20250115 14:51:53.085188 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a6b76cb343fb4456ac07fd3c6e9888de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6b76cb343fb4456ac07fd3c6e9888de already in progress: creating tablet
I20250115 14:51:53.088160 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet be18f1b6d8a24fabbe818811ef646c62 (table default.Table_40664b92268d4ba39f3f5852835f97d2 [id=da39f1615db74469824c3bf8eb74d260]) successfully deleted
I20250115 14:51:53.088433 1164 tablet_bootstrap.cc:492] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:53.088718 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd72a0d586d4a2c817f6a39daa07063 with delete_type TABLET_DATA_DELETED (replaced by tablet d7309ecb11d844848638c042c9bbe5f1) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.089987 1165 tablet_replica.cc:331] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.090467 1165 raft_consensus.cc:2238] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:53.091028 1165 pending_rounds.cc:62] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:53.091632 1165 pending_rounds.cc:69] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558311496503296 op_type: NO_OP noop_request { }
I20250115 14:51:53.092001 1165 raft_consensus.cc:2883] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:53.092278 1165 raft_consensus.cc:2267] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.091485 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b76cb343fb4456ac07fd3c6e9888de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.089618 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 24751e606ad94675b650d67a3738481e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:53.093461 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 58cca6072e4e4f608c6a45be279304f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58cca6072e4e4f608c6a45be279304f8 already in progress: creating tablet
I20250115 14:51:53.093662 1165 ts_tablet_manager.cc:1905] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:53.095438 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3ae60842a2d4486eb7651c3f710ee1e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ae60842a2d4486eb7651c3f710ee1e8 already in progress: creating tablet
W20250115 14:51:53.096036 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3fd72a0d586d4a2c817f6a39daa07063 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd72a0d586d4a2c817f6a39daa07063 already in progress: creating tablet
W20250115 14:51:53.096657 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 24751e606ad94675b650d67a3738481e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24751e606ad94675b650d67a3738481e already in progress: creating tablet
W20250115 14:51:53.097229 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a6b76cb343fb4456ac07fd3c6e9888de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6b76cb343fb4456ac07fd3c6e9888de already in progress: creating tablet
I20250115 14:51:53.099471 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3a38a6bc32314396b497e7ae166a63f7
I20250115 14:51:53.102783 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a38a6bc32314396b497e7ae166a63f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.103338 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a38a6bc32314396b497e7ae166a63f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.103824 1167 tablet_replica.cc:331] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.104043 1167 raft_consensus.cc:2238] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.104223 1167 raft_consensus.cc:2267] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.105090 1162 tablet_bootstrap.cc:492] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.105079 1167 ts_tablet_manager.cc:1905] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.105311 1162 ts_tablet_manager.cc:1397] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.126s user 0.001s sys 0.009s
I20250115 14:51:53.105779 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c06f4c91e37e4b6f95521174e05efb24" } modify_external_catalogs: true
I20250115 14:51:53.105981 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c06f4c91e37e4b6f95521174e05efb24" } modify_external_catalogs: true
I20250115 14:51:53.106277 1165 ts_tablet_manager.cc:1918] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:53.106410 1165 log.cc:1198] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/66c6c392518a464dab664e5f9321f1f6
I20250115 14:51:53.106772 1165 ts_tablet_manager.cc:1939] T 66c6c392518a464dab664e5f9321f1f6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.107316 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ff3a0751d563485db14fc5a44116f2ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.107950 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff3a0751d563485db14fc5a44116f2ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.109135 1165 tablet_replica.cc:331] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.109330 1165 raft_consensus.cc:2238] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.109236 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d9be1b09a8294cb4bf71098ae4c97b80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:53.109467 1165 raft_consensus.cc:2267] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:53.109747 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9be1b09a8294cb4bf71098ae4c97b80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.110558 1165 ts_tablet_manager.cc:1905] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.110867 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_85673afa950047b1aee61fc1b7692be3" } new_table_name: "default.Table_76ab87d8a84c47008813112669e3800b" modify_external_catalogs: true
I20250115 14:51:53.111994 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 66c6c392518a464dab664e5f9321f1f6 (table default.Table_7281d9e98ae843dc915b4932d3d7d50b [id=52b6d8da69f74139a394cafe66e48a35]) successfully deleted
I20250115 14:51:53.112399 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c00f03edc74af8b084bb236487a347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:53.113896 1221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250115 14:51:53.114064 1221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 14:51:53.114151 1221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250115 14:51:53.114238 1221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250115 14:51:53.114341 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a38a6bc32314396b497e7ae166a63f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:53.115015 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3a38a6bc32314396b497e7ae166a63f7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a38a6bc32314396b497e7ae166a63f7 already in progress: creating tablet
I20250115 14:51:53.116041 1164 tablet_bootstrap.cc:654] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.116832 1167 ts_tablet_manager.cc:1918] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.117103 1167 log.cc:1198] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/3a38a6bc32314396b497e7ae166a63f7
I20250115 14:51:53.117537 1167 ts_tablet_manager.cc:1939] T 3a38a6bc32314396b497e7ae166a63f7 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.106333 1162 raft_consensus.cc:357] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.120540 1162 raft_consensus.cc:383] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.120638 1162 raft_consensus.cc:738] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.120901 1162 consensus_queue.cc:260] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.121460 1162 ts_tablet_manager.cc:1428] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250115 14:51:53.121819 1162 tablet_bootstrap.cc:492] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.121923 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ea0f104c1a71404591072e7a36b276b0
I20250115 14:51:53.122431 452 catalog_manager.cc:6596] ReplaceTablet: tablet ea0f104c1a71404591072e7a36b276b0 deleted and replaced by tablet b3f07ba1d929468eb8eb1f949e9931b9
I20250115 14:51:53.125283 1168 tablet_bootstrap.cc:492] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.125451 1168 ts_tablet_manager.cc:1397] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.153s user 0.029s sys 0.064s
I20250115 14:51:53.126040 1162 tablet_bootstrap.cc:654] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.126678 1168 raft_consensus.cc:357] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.126900 1168 raft_consensus.cc:383] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.126966 1168 raft_consensus.cc:738] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.127192 1168 consensus_queue.cc:260] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.127782 1168 ts_tablet_manager.cc:1428] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:53.128559 1168 tablet_bootstrap.cc:492] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.130407 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f104c1a71404591072e7a36b276b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b3f07ba1d929468eb8eb1f949e9931b9) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.130883 889 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f104c1a71404591072e7a36b276b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b3f07ba1d929468eb8eb1f949e9931b9) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.131001 1168 tablet_bootstrap.cc:654] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.132097 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_48222c861a6e490a82607bb753a21cf6" } new_table_name: "default.Table_26cfaab25925425bb14d5384a6ce1827" modify_external_catalogs: true
I20250115 14:51:53.133284 1165 ts_tablet_manager.cc:1918] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.133414 1165 log.cc:1198] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f224f857b1644596853acd9e3a2a4a5f
I20250115 14:51:53.134357 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0dbb36cfea4d45d88c5ef963c153cf77" } new_table_name: "default.Table_f88c333bf687464cb16fb61569488e70" modify_external_catalogs: true
I20250115 14:51:53.134285 1165 ts_tablet_manager.cc:1939] T f224f857b1644596853acd9e3a2a4a5f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.135560 1165 tablet_replica.cc:331] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.135593 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2752e5f3a4ca4c41a614738b9354d962" } modify_external_catalogs: true
I20250115 14:51:53.135739 1165 raft_consensus.cc:2238] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.135773 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2752e5f3a4ca4c41a614738b9354d962" } modify_external_catalogs: true
I20250115 14:51:53.135890 1165 raft_consensus.cc:2267] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.136416 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fbd024b67b794f99b66433ffbb89184c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:53.136636 1165 ts_tablet_manager.cc:1905] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:53.138227 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbd024b67b794f99b66433ffbb89184c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.129858 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 3a38a6bc32314396b497e7ae166a63f7 (table default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]) successfully deleted
I20250115 14:51:53.139779 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f224f857b1644596853acd9e3a2a4a5f (table default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]) successfully deleted
W20250115 14:51:53.139992 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ea0f104c1a71404591072e7a36b276b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0f104c1a71404591072e7a36b276b0 already in progress: creating tablet
I20250115 14:51:53.140457 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 74bfda3861d148bc8b67ea071909d22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.140791 1221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4191
ASAN enabled
I20250115 14:51:53.143230 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_59fb977786f045519df5d218a7fd5e8d" } new_table_name: "default.Table_d5ec136542e043e99db4296e025d725f" modify_external_catalogs: true
I20250115 14:51:53.143898 1221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 14:51:53.122740 753 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f104c1a71404591072e7a36b276b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b3f07ba1d929468eb8eb1f949e9931b9) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:53.145229 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ea0f104c1a71404591072e7a36b276b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0f104c1a71404591072e7a36b276b0 already in progress: creating tablet
I20250115 14:51:53.146379 1162 tablet_bootstrap.cc:492] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.146600 1162 ts_tablet_manager.cc:1397] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.001s
I20250115 14:51:53.147464 1162 raft_consensus.cc:357] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.147790 1162 raft_consensus.cc:383] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.147909 1162 raft_consensus.cc:738] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.148800 1221 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 14:51:53.148861 1168 tablet_bootstrap.cc:492] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.148947 1162 consensus_queue.cc:260] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.149067 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 3fd72a0d586d4a2c817f6a39daa07063 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.149348 1168 ts_tablet_manager.cc:1397] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20250115 14:51:53.149657 1162 ts_tablet_manager.cc:1428] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250115 14:51:53.150039 1162 tablet_bootstrap.cc:492] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.150516 1168 raft_consensus.cc:357] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.151032 1168 raft_consensus.cc:383] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.151126 1168 raft_consensus.cc:738] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.151521 1168 consensus_queue.cc:260] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.152040 1168 ts_tablet_manager.cc:1428] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250115 14:51:53.152343 1168 tablet_bootstrap.cc:492] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.155503 1162 tablet_bootstrap.cc:654] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.157130 1165 ts_tablet_manager.cc:1918] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.157249 1165 log.cc:1198] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e115bdfa1d69421890122f9afdc23b29
I20250115 14:51:53.157678 1165 ts_tablet_manager.cc:1939] T e115bdfa1d69421890122f9afdc23b29 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.158169 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1230ddf0f50b4c88ae6762341fb2b04a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:53.158592 1168 tablet_bootstrap.cc:654] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:53.158684 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1230ddf0f50b4c88ae6762341fb2b04a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.161283 1165 tablet_replica.cc:331] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.161484 1165 raft_consensus.cc:2238] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.161618 1165 raft_consensus.cc:2267] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.162616 1165 ts_tablet_manager.cc:1905] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.163749 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e115bdfa1d69421890122f9afdc23b29 (table default.Table_2752e5f3a4ca4c41a614738b9354d962 [id=1c779c8a53f54707abe1d619b30a912e]) successfully deleted
I20250115 14:51:53.163918 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 6326b1526191464485855b657e34a2b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:53.164011 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 58cca6072e4e4f608c6a45be279304f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58cca6072e4e4f608c6a45be279304f8 already in progress: creating tablet
W20250115 14:51:53.164230 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 3ae60842a2d4486eb7651c3f710ee1e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ae60842a2d4486eb7651c3f710ee1e8 already in progress: creating tablet
W20250115 14:51:53.164849 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 3fd72a0d586d4a2c817f6a39daa07063 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fd72a0d586d4a2c817f6a39daa07063 already in progress: creating tablet
W20250115 14:51:53.165140 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a6b76cb343fb4456ac07fd3c6e9888de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6b76cb343fb4456ac07fd3c6e9888de already in progress: creating tablet
W20250115 14:51:53.165405 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 24751e606ad94675b650d67a3738481e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24751e606ad94675b650d67a3738481e already in progress: creating tablet
I20250115 14:51:53.170873 1165 ts_tablet_manager.cc:1918] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.171018 1165 log.cc:1198] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3a38a6bc32314396b497e7ae166a63f7
I20250115 14:51:53.171309 1165 ts_tablet_manager.cc:1939] T 3a38a6bc32314396b497e7ae166a63f7 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:51:53.171744 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d77ff794a83348c29391786666987db1 consensus state has no leader
I20250115 14:51:53.173416 1162 tablet_bootstrap.cc:492] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.173553 1162 ts_tablet_manager.cc:1397] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.004s
I20250115 14:51:53.173429 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_aa6a106222694bbfbe6c26b0f8e65396"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.173959 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa6a106222694bbfbe6c26b0f8e65396 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.174301 1162 raft_consensus.cc:357] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.174463 1168 tablet_bootstrap.cc:492] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.174484 1162 raft_consensus.cc:383] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.174559 1162 raft_consensus.cc:738] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.174616 1168 ts_tablet_manager.cc:1397] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20250115 14:51:53.174767 1162 consensus_queue.cc:260] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.175269 1162 ts_tablet_manager.cc:1428] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:53.175388 1168 raft_consensus.cc:357] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.175572 1162 tablet_bootstrap.cc:492] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.175581 1168 raft_consensus.cc:383] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.175643 1168 raft_consensus.cc:738] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.175868 1168 consensus_queue.cc:260] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.177971 1168 ts_tablet_manager.cc:1428] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:53.178251 1168 tablet_bootstrap.cc:492] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.179540 1162 tablet_bootstrap.cc:654] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:53.179622 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7a9e2c053d624b6ba71b031a591475a4 has no consensus state
I20250115 14:51:53.180876 1168 tablet_bootstrap.cc:654] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.182659 1168 tablet_bootstrap.cc:492] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.182790 1168 ts_tablet_manager.cc:1397] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:51:53.183454 1168 raft_consensus.cc:357] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.183620 1168 raft_consensus.cc:383] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.183710 1168 raft_consensus.cc:738] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.183929 1168 consensus_queue.cc:260] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.185199 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]). Expected version 1 got 0
I20250115 14:51:53.185410 454 catalog_manager.cc:5526] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:53.185863 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]). Expected version 2 got 0
I20250115 14:51:53.186090 454 catalog_manager.cc:5526] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:53.186352 1168 ts_tablet_manager.cc:1428] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250115 14:51:53.186617 1168 tablet_bootstrap.cc:492] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.188156 1165 tablet_replica.cc:331] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.188397 1165 raft_consensus.cc:2238] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.188577 1165 raft_consensus.cc:2267] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:53.189074 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb792ca1b3f54def92f504ec8682d2c7 consensus state has no leader
I20250115 14:51:53.189092 1162 tablet_bootstrap.cc:492] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.189237 1162 ts_tablet_manager.cc:1397] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20250115 14:51:53.189971 1165 ts_tablet_manager.cc:1905] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.190068 1162 raft_consensus.cc:357] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.190276 1162 raft_consensus.cc:383] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.190346 1162 raft_consensus.cc:738] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.190603 1162 consensus_queue.cc:260] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.191138 1162 ts_tablet_manager.cc:1428] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250115 14:51:53.191426 1162 tablet_bootstrap.cc:492] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.191676 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeca9975f1945ad956f47db1ea16a73 (DEFAULT_TABLE table=default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666 [id=914642e9e7974554a76a94e205db108f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.192097 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aeca9975f1945ad956f47db1ea16a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.193109 1162 tablet_bootstrap.cc:654] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.196049 1165 ts_tablet_manager.cc:1918] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.196148 1165 log.cc:1198] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/13c00f03edc74af8b084bb236487a347
I20250115 14:51:53.196619 751 tablet_service.cc:1467] Processing CreateTablet for tablet 83097c5bfe364418a213236435edc40c (DEFAULT_TABLE table=default.Table_9126b20cb8474e8e9817459ddf0b9b0c [id=caa6adb7563c41088e5120993c46c54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.196902 1168 tablet_bootstrap.cc:654] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.197027 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83097c5bfe364418a213236435edc40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.199982 1165 ts_tablet_manager.cc:1939] T 13c00f03edc74af8b084bb236487a347 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.200596 754 tablet_service.cc:1467] Processing CreateTablet for tablet b25dce523e724066b203e6111cad8111 (DEFAULT_TABLE table=default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.200979 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25dce523e724066b203e6111cad8111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.201356 1162 tablet_bootstrap.cc:492] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.201490 1162 ts_tablet_manager.cc:1397] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20250115 14:51:53.202970 1165 tablet_replica.cc:331] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.202931 1162 raft_consensus.cc:357] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.203154 1162 raft_consensus.cc:383] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.203241 1162 raft_consensus.cc:738] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.203470 1162 consensus_queue.cc:260] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.203735 1168 tablet_bootstrap.cc:492] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.203883 1168 ts_tablet_manager.cc:1397] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20250115 14:51:53.203974 1162 ts_tablet_manager.cc:1428] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:53.204214 1162 tablet_bootstrap.cc:492] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.204169 741 tablet_service.cc:1467] Processing CreateTablet for tablet 1839edd0d49b467d8bb31b3001ff89eb (DEFAULT_TABLE table=default.Table_c368b4ef34f147b6984aee76adde3cf7 [id=a724f6e4ad16485b8ee2445bb072be3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.204551 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1839edd0d49b467d8bb31b3001ff89eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.204597 1168 raft_consensus.cc:357] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.204809 1168 raft_consensus.cc:383] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.204878 1168 raft_consensus.cc:738] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.205109 1168 consensus_queue.cc:260] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.206061 1162 tablet_bootstrap.cc:654] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.206733 1168 ts_tablet_manager.cc:1428] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:53.207024 1168 tablet_bootstrap.cc:492] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.208597 749 tablet_service.cc:1467] Processing CreateTablet for tablet d5a5a92804cb462181e68e1cec02102b (DEFAULT_TABLE table=default.Table_c0256da787104a42a1bc1e3946e11fd6 [id=22c418c3ac6648c687a8ae14d439aea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.208997 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a5a92804cb462181e68e1cec02102b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.209164 1168 tablet_bootstrap.cc:654] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.203162 1165 raft_consensus.cc:2238] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.210121 1165 raft_consensus.cc:2267] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.210683 1165 ts_tablet_manager.cc:1905] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.212385 752 tablet_service.cc:1467] Processing CreateTablet for tablet e50462626f914fb1b0f440c48f30e72d (DEFAULT_TABLE table=default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.212733 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e50462626f914fb1b0f440c48f30e72d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:53.213164 1324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:51:53.215056 1162 tablet_bootstrap.cc:492] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.215314 1162 ts_tablet_manager.cc:1397] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250115 14:51:53.217111 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c7fd75443d3a4d108674df939f2c67f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.217689 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7fd75443d3a4d108674df939f2c67f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.218218 1165 ts_tablet_manager.cc:1918] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.218344 1165 log.cc:1198] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/74bfda3861d148bc8b67ea071909d22d
I20250115 14:51:53.218680 1165 ts_tablet_manager.cc:1939] T 74bfda3861d148bc8b67ea071909d22d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.220671 1168 tablet_bootstrap.cc:492] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.220834 1168 ts_tablet_manager.cc:1397] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250115 14:51:53.220898 890 tablet_service.cc:1467] Processing CreateTablet for tablet 83097c5bfe364418a213236435edc40c (DEFAULT_TABLE table=default.Table_9126b20cb8474e8e9817459ddf0b9b0c [id=caa6adb7563c41088e5120993c46c54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.221002 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeca9975f1945ad956f47db1ea16a73 (DEFAULT_TABLE table=default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666 [id=914642e9e7974554a76a94e205db108f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.221195 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 83097c5bfe364418a213236435edc40c (DEFAULT_TABLE table=default.Table_9126b20cb8474e8e9817459ddf0b9b0c [id=caa6adb7563c41088e5120993c46c54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.221336 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83097c5bfe364418a213236435edc40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.221350 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aeca9975f1945ad956f47db1ea16a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.221570 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83097c5bfe364418a213236435edc40c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.221735 1168 raft_consensus.cc:357] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.221926 1168 raft_consensus.cc:383] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.222019 1168 raft_consensus.cc:738] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.222368 1168 consensus_queue.cc:260] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.223791 1168 ts_tablet_manager.cc:1428] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:53.224067 1168 tablet_bootstrap.cc:492] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.224327 879 tablet_service.cc:1467] Processing CreateTablet for tablet 1839edd0d49b467d8bb31b3001ff89eb (DEFAULT_TABLE table=default.Table_c368b4ef34f147b6984aee76adde3cf7 [id=a724f6e4ad16485b8ee2445bb072be3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.224597 1018 tablet_service.cc:1467] Processing CreateTablet for tablet b25dce523e724066b203e6111cad8111 (DEFAULT_TABLE table=default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.224701 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1839edd0d49b467d8bb31b3001ff89eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.224947 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25dce523e724066b203e6111cad8111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.226485 1168 tablet_bootstrap.cc:654] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.227458 1165 tablet_replica.cc:331] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.227635 1165 raft_consensus.cc:2238] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.227818 1165 raft_consensus.cc:2267] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.228433 1165 ts_tablet_manager.cc:1905] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.220919 891 tablet_service.cc:1467] Processing CreateTablet for tablet b25dce523e724066b203e6111cad8111 (DEFAULT_TABLE table=default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.228806 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25dce523e724066b203e6111cad8111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.228936 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3a38a6bc32314396b497e7ae166a63f7 (table default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]) successfully deleted
I20250115 14:51:53.229168 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 13c00f03edc74af8b084bb236487a347 (table default.Table_64abfdac91a54d6bb59ccfc7791a170c [id=358f8fad6d944439847ea4ce82f1213a]) successfully deleted
W20250115 14:51:53.229394 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ea0f104c1a71404591072e7a36b276b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea0f104c1a71404591072e7a36b276b0 already in progress: creating tablet
I20250115 14:51:53.229549 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 74bfda3861d148bc8b67ea071909d22d (table default.Table_a78c10f9e6384d7d89d3add6f1371429 [id=2c59ffcdee944c4289ad5dd38dc2e8cc]) successfully deleted
I20250115 14:51:53.230319 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 1839edd0d49b467d8bb31b3001ff89eb (DEFAULT_TABLE table=default.Table_c368b4ef34f147b6984aee76adde3cf7 [id=a724f6e4ad16485b8ee2445bb072be3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.230661 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1839edd0d49b467d8bb31b3001ff89eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.231061 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d5a5a92804cb462181e68e1cec02102b (DEFAULT_TABLE table=default.Table_c0256da787104a42a1bc1e3946e11fd6 [id=22c418c3ac6648c687a8ae14d439aea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.231393 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a5a92804cb462181e68e1cec02102b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.220640 889 tablet_service.cc:1467] Processing CreateTablet for tablet 9aeca9975f1945ad956f47db1ea16a73 (DEFAULT_TABLE table=default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666 [id=914642e9e7974554a76a94e205db108f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.234344 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aeca9975f1945ad956f47db1ea16a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.235167 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_18aff4b27c7846fbaede10b093240587" } new_table_name: "default.Table_9a090445edb041279a25064152cecee2" modify_external_catalogs: true
I20250115 14:51:53.237811 884 tablet_service.cc:1467] Processing CreateTablet for tablet d5a5a92804cb462181e68e1cec02102b (DEFAULT_TABLE table=default.Table_c0256da787104a42a1bc1e3946e11fd6 [id=22c418c3ac6648c687a8ae14d439aea5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.238252 885 tablet_service.cc:1467] Processing CreateTablet for tablet e50462626f914fb1b0f440c48f30e72d (DEFAULT_TABLE table=default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.241197 747 tablet_service.cc:1467] Processing CreateTablet for tablet 6b60ff80679a470cb4bdced71cc14195 (DEFAULT_TABLE table=default.Table_910803aa49a042acba7e5a6a65b2e85d [id=201c214e2eab4873af2095e83dc88b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.241617 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b60ff80679a470cb4bdced71cc14195. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.242662 756 tablet_service.cc:1467] Processing CreateTablet for tablet 3e990a2920b542f69a8bdd022db6e7e2 (DEFAULT_TABLE table=default.Table_b17b7ebacc9a4d32b756eff66de640dd [id=e6177a285e0c4339a6cfc4a0ae779eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.243009 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e990a2920b542f69a8bdd022db6e7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.244771 888 tablet_service.cc:1467] Processing CreateTablet for tablet 6b60ff80679a470cb4bdced71cc14195 (DEFAULT_TABLE table=default.Table_910803aa49a042acba7e5a6a65b2e85d [id=201c214e2eab4873af2095e83dc88b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.244892 880 tablet_service.cc:1467] Processing CreateTablet for tablet d86075ae5b894d5986ac4bd538f4d42a (DEFAULT_TABLE table=default.Table_f7c3d078ce8a49cfa5965cb29b102e42 [id=54e9cab363084972abe02521b2cc19b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.245757 886 tablet_service.cc:1467] Processing CreateTablet for tablet 3e990a2920b542f69a8bdd022db6e7e2 (DEFAULT_TABLE table=default.Table_b17b7ebacc9a4d32b756eff66de640dd [id=e6177a285e0c4339a6cfc4a0ae779eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.246376 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5a5a92804cb462181e68e1cec02102b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.247726 1019 tablet_service.cc:1467] Processing CreateTablet for tablet e50462626f914fb1b0f440c48f30e72d (DEFAULT_TABLE table=default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.251003 1165 ts_tablet_manager.cc:1918] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.251019 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 6b60ff80679a470cb4bdced71cc14195 (DEFAULT_TABLE table=default.Table_910803aa49a042acba7e5a6a65b2e85d [id=201c214e2eab4873af2095e83dc88b23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.251564 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e50462626f914fb1b0f440c48f30e72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.251775 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b60ff80679a470cb4bdced71cc14195. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.252178 1021 tablet_service.cc:1467] Processing CreateTablet for tablet d86075ae5b894d5986ac4bd538f4d42a (DEFAULT_TABLE table=default.Table_f7c3d078ce8a49cfa5965cb29b102e42 [id=54e9cab363084972abe02521b2cc19b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.242410 746 tablet_service.cc:1467] Processing CreateTablet for tablet d86075ae5b894d5986ac4bd538f4d42a (DEFAULT_TABLE table=default.Table_f7c3d078ce8a49cfa5965cb29b102e42 [id=54e9cab363084972abe02521b2cc19b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.252250 1162 raft_consensus.cc:357] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.252638 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86075ae5b894d5986ac4bd538f4d42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.254702 1165 log.cc:1198] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6326b1526191464485855b657e34a2b9
I20250115 14:51:53.254920 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3e990a2920b542f69a8bdd022db6e7e2 (DEFAULT_TABLE table=default.Table_b17b7ebacc9a4d32b756eff66de640dd [id=e6177a285e0c4339a6cfc4a0ae779eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.257601 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b60ff80679a470cb4bdced71cc14195. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.258479 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e990a2920b542f69a8bdd022db6e7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.259987 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86075ae5b894d5986ac4bd538f4d42a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:53.261905 1090 debug-util.cc:398] Leaking SignalData structure 0x60300086d400 after lost signal to thread 1236
I20250115 14:51:53.263357 1165 ts_tablet_manager.cc:1939] T 6326b1526191464485855b657e34a2b9 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.264357 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d86075ae5b894d5986ac4bd538f4d42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.264683 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e50462626f914fb1b0f440c48f30e72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.266062 1162 raft_consensus.cc:383] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.270885 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6326b1526191464485855b657e34a2b9 (table default.Table_662e371503c24e6cacf91cb2afe19d53 [id=ade9599dbabb4c298debe2172859adcc]) successfully deleted
I20250115 14:51:53.273772 1162 raft_consensus.cc:738] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.274268 1162 consensus_queue.cc:260] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.274889 1162 ts_tablet_manager.cc:1428] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20250115 14:51:53.275936 1162 tablet_bootstrap.cc:492] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.275962 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e990a2920b542f69a8bdd022db6e7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.276209 1169 raft_consensus.cc:491] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:51:53.276795 1322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:51:53.279881 1328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:51:53.281766 1221 server_base.cc:1034] running on GCE node
I20250115 14:51:53.282951 1221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 14:51:53.280164 1169 raft_consensus.cc:513] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.290882 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d43cd66b18f4649894a629637037822" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.293808 1162 tablet_bootstrap.cc:654] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.293884 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d43cd66b18f4649894a629637037822" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:53.294142 912 raft_consensus.cc:2463] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:53.294612 983 leader_election.cc:304] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:53.294958 1169 leader_election.cc:290] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:53.295276 1225 raft_consensus.cc:2798] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:53.295385 1225 raft_consensus.cc:491] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:53.295831 1225 raft_consensus.cc:3054] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.296540 1221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20250115 14:51:53.300542 984 leader_election.cc:343] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.304020 1221 hybrid_clock.cc:648] HybridClock initialized: now 1736952713303844 us; error 148 us; skew 500 ppm
I20250115 14:51:53.304710 1221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 14:51:53.305024 1164 tablet_bootstrap.cc:492] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:53.305179 1164 ts_tablet_manager.cc:1397] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.217s user 0.003s sys 0.000s
I20250115 14:51:53.307011 1164 raft_consensus.cc:357] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.307216 1164 raft_consensus.cc:383] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.307302 1164 raft_consensus.cc:738] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.307520 1164 consensus_queue.cc:260] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.308022 1164 ts_tablet_manager.cc:1428] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:51:53.309899 1164 tablet_bootstrap.cc:492] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:53.311547 1164 tablet_bootstrap.cc:654] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.314289 1225 raft_consensus.cc:513] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.315590 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d43cd66b18f4649894a629637037822" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:53.315834 912 raft_consensus.cc:3054] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.318804 912 raft_consensus.cc:2463] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:53.319859 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d43cd66b18f4649894a629637037822" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:53.322429 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d3300e90dc3845f78b3c63dfbf681dde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.323037 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3300e90dc3845f78b3c63dfbf681dde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.323087 983 leader_election.cc:304] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:51:53.323382 984 leader_election.cc:343] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.323884 1169 raft_consensus.cc:2798] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:53.324024 1169 raft_consensus.cc:695] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:53.324318 1169 consensus_queue.cc:237] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:53.334410 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e3ec000089a47a2930dec382192cd6f consensus state has no leader
I20250115 14:51:53.334477 1225 leader_election.cc:290] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:53.334728 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 58cca6072e4e4f608c6a45be279304f8 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.335395 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_620e744402d54372bb5c38ad4e168dc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.335968 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_620e744402d54372bb5c38ad4e168dc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.337095 1209 raft_consensus.cc:491] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.337239 449 master_service.cc:946] ReplaceTablet: received request to replace tablet a6b76cb343fb4456ac07fd3c6e9888de from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.337270 1209 raft_consensus.cc:513] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.337755 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_30d9bde332c24d2a94267ffd2605a7cf" } modify_external_catalogs: true
I20250115 14:51:53.337937 1209 leader_election.cc:290] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.337940 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_30d9bde332c24d2a94267ffd2605a7cf" } modify_external_catalogs: true
I20250115 14:51:53.338534 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf909c84eb314d95b73b6de82aebe9f2" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:53.338788 1049 raft_consensus.cc:2463] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:53.339377 847 leader_election.cc:304] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:53.340065 1209 raft_consensus.cc:2798] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:53.340155 1209 raft_consensus.cc:491] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:53.340260 1209 raft_consensus.cc:3054] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.340579 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_26cfaab25925425bb14d5384a6ce1827" } modify_external_catalogs: true
I20250115 14:51:53.340728 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_26cfaab25925425bb14d5384a6ce1827" } modify_external_catalogs: true
I20250115 14:51:53.341976 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d243daba84ff4befa0c4e7bac5fe7476
I20250115 14:51:53.342459 450 master_service.cc:946] ReplaceTablet: received request to replace tablet d4fb2d33f3c745beb4a4ba036e23a8e0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.343003 1209 raft_consensus.cc:513] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.343534 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f312f1e3a33a404ca7284c713859f156
I20250115 14:51:53.344097 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d9be1b09a8294cb4bf71098ae4c97b80" } modify_external_catalogs: true
I20250115 14:51:53.344249 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d9be1b09a8294cb4bf71098ae4c97b80" } modify_external_catalogs: true
I20250115 14:51:53.344283 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf909c84eb314d95b73b6de82aebe9f2" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:53.344501 1049 raft_consensus.cc:3054] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.346149 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_12bf6210790a443dbd238713e2f8a874"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.346693 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12bf6210790a443dbd238713e2f8a874 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.347258 1049 raft_consensus.cc:2463] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
W20250115 14:51:53.347782 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ab1a38186c5042ff851f57979ab80dbc consensus state has no leader
I20250115 14:51:53.348042 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 5d51e53b4ffa45bfbb9553d7a5870844 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.348140 847 leader_election.cc:304] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:53.348573 1207 raft_consensus.cc:2798] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:53.348738 1207 raft_consensus.cc:695] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:53.348997 1207 consensus_queue.cc:237] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.349241 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]). Expected version 1 got 0
I20250115 14:51:53.349936 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dc004b8b6d6944e6a55197bf7065f0f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.350405 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc004b8b6d6944e6a55197bf7065f0f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.352753 754 tablet_service.cc:1514] Processing DeleteTablet for tablet d243daba84ff4befa0c4e7bac5fe7476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.352921 1209 leader_election.cc:290] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.353132 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf909c84eb314d95b73b6de82aebe9f2" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.353425 752 tablet_service.cc:1514] Processing DeleteTablet for tablet f312f1e3a33a404ca7284c713859f156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.353504 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf909c84eb314d95b73b6de82aebe9f2" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:53.353755 848 leader_election.cc:343] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:53.354056 848 leader_election.cc:343] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.354224 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d5e0fb344ffa468286ab7ab95c01b841"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.354658 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5e0fb344ffa468286ab7ab95c01b841 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:53.354635 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d243daba84ff4befa0c4e7bac5fe7476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d243daba84ff4befa0c4e7bac5fe7476 already in progress: creating tablet
W20250115 14:51:53.354943 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f312f1e3a33a404ca7284c713859f156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f312f1e3a33a404ca7284c713859f156 already in progress: creating tablet
I20250115 14:51:53.356473 749 tablet_service.cc:1514] Processing DeleteTablet for tablet f8346c87744f46daa623c414e7307299 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.356886 1166 tablet_replica.cc:331] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:53.357074 1166 raft_consensus.cc:2238] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.357232 1166 raft_consensus.cc:2267] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.357985 1166 ts_tablet_manager.cc:1905] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.358991 1168 tablet_bootstrap.cc:492] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.359133 1168 ts_tablet_manager.cc:1397] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.135s user 0.040s sys 0.045s
I20250115 14:51:53.360903 1168 raft_consensus.cc:357] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.361109 1168 raft_consensus.cc:383] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.361186 1168 raft_consensus.cc:738] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.362345 1168 consensus_queue.cc:260] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.365911 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a38a6bc32314396b497e7ae166a63f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.367244 1168 ts_tablet_manager.cc:1428] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:51:53.367431 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet f960c88df14242c5ada1ecf9d8e68196 (table default.Table_56a365aae805419683d1f4dfd97951cc [id=24d4e203bce64fcdbd5b6fa30a99e6b2]). Expected version 1 got 0
I20250115 14:51:53.367730 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d6a9fb932c5b4545a4609d568f0e5494 (table default.Table_8f6e7d6b890e4a449f170811c030b6f5 [id=3f3fecceceb3464bb12462bb3041b458]). Expected version 2 got 0
I20250115 14:51:53.367937 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_881f3ef58e824b4cb4e1fb3ec4846cc7 [id=0ef6f07cf29e46e69bf2a4449187a4e7]). Expected version 1 got 0
I20250115 14:51:53.367686 1168 tablet_bootstrap.cc:492] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.368250 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 6326b1526191464485855b657e34a2b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.368507 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d243daba84ff4befa0c4e7bac5fe7476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.368752 890 tablet_service.cc:1514] Processing DeleteTablet for tablet f312f1e3a33a404ca7284c713859f156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:53.368749 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3a38a6bc32314396b497e7ae166a63f7 because the tablet was not found. No further retry: Not found: Tablet not found: 3a38a6bc32314396b497e7ae166a63f7
I20250115 14:51:53.368988 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d43cd66b18f4649894a629637037822 with delete_type TABLET_DATA_DELETED (replaced by tablet bb792ca1b3f54def92f504ec8682d2c7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.370079 1167 tablet_replica.cc:331] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.370260 1167 raft_consensus.cc:2238] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.370429 1167 raft_consensus.cc:2267] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.370884 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet d243daba84ff4befa0c4e7bac5fe7476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.370959 1167 ts_tablet_manager.cc:1905] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.369259 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f312f1e3a33a404ca7284c713859f156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.371095 1166 ts_tablet_manager.cc:1918] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.371251 1166 log.cc:1198] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f8346c87744f46daa623c414e7307299
I20250115 14:51:53.371207 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bbe00fc656b44633ae2fd159ae77ffc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:53.371582 1166 ts_tablet_manager.cc:1939] T f8346c87744f46daa623c414e7307299 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:51:53.371768 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbe00fc656b44633ae2fd159ae77ffc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.373505 880 tablet_service.cc:1514] Processing DeleteTablet for tablet ad67354f35ce4564b88dbdffcabf3564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.374116 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f8346c87744f46daa623c414e7307299 (table default.Table_18add812314a4b0f97f9a61776eb768a [id=c85266bab3cd4586a91a82c1f7f96c09]) successfully deleted
I20250115 14:51:53.376814 1162 tablet_bootstrap.cc:492] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.376991 1162 ts_tablet_manager.cc:1397] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.010s
I20250115 14:51:53.378346 1162 raft_consensus.cc:357] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.378571 1162 raft_consensus.cc:383] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.378654 1162 raft_consensus.cc:738] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.378955 1162 consensus_queue.cc:260] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:53.379330 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d243daba84ff4befa0c4e7bac5fe7476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d243daba84ff4befa0c4e7bac5fe7476 already in progress: creating tablet
I20250115 14:51:53.380405 1162 ts_tablet_manager.cc:1428] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20250115 14:51:53.380703 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f312f1e3a33a404ca7284c713859f156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f312f1e3a33a404ca7284c713859f156 already in progress: creating tablet
I20250115 14:51:53.380748 1162 tablet_bootstrap.cc:492] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.380411 1167 ts_tablet_manager.cc:1918] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.381028 1167 log.cc:1198] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/6326b1526191464485855b657e34a2b9
I20250115 14:51:53.381430 1167 ts_tablet_manager.cc:1939] T 6326b1526191464485855b657e34a2b9 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.382455 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 6326b1526191464485855b657e34a2b9 (table default.Table_662e371503c24e6cacf91cb2afe19d53 [id=ade9599dbabb4c298debe2172859adcc]) successfully deleted
W20250115 14:51:53.383072 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d243daba84ff4befa0c4e7bac5fe7476 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d243daba84ff4befa0c4e7bac5fe7476 already in progress: creating tablet
I20250115 14:51:53.384119 1221 webserver.cc:458] Webserver started at http://127.0.104.60:37445/ using document root <none> and password file <none>
I20250115 14:51:53.384336 1162 tablet_bootstrap.cc:654] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:53.384734 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f312f1e3a33a404ca7284c713859f156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f312f1e3a33a404ca7284c713859f156 already in progress: creating tablet
I20250115 14:51:53.385087 1167 tablet_replica.cc:331] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.385267 1221 fs_manager.cc:362] Metadata directory not provided
I20250115 14:51:53.385322 1167 raft_consensus.cc:2238] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.385437 1221 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 14:51:53.385612 1167 raft_consensus.cc:2267] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.386123 1168 tablet_bootstrap.cc:654] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.386895 1167 ts_tablet_manager.cc:1905] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.389896 1164 tablet_bootstrap.cc:492] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:53.390082 1164 ts_tablet_manager.cc:1397] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.000s
I20250115 14:51:53.391374 1164 raft_consensus.cc:357] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.391589 1164 raft_consensus.cc:383] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.391664 1164 raft_consensus.cc:738] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.391925 1164 consensus_queue.cc:260] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.392421 1164 ts_tablet_manager.cc:1428] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250115 14:51:53.394151 1168 tablet_bootstrap.cc:492] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.394519 1168 ts_tablet_manager.cc:1397] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.006s
I20250115 14:51:53.395258 1168 raft_consensus.cc:357] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.395444 1168 raft_consensus.cc:383] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.395525 1168 raft_consensus.cc:738] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.395550 1221 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.001s
I20250115 14:51:53.395773 1168 consensus_queue.cc:260] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.396557 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b3f07ba1d929468eb8eb1f949e9931b9
I20250115 14:51:53.399920 1168 ts_tablet_manager.cc:1428] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:53.400063 1164 tablet_bootstrap.cc:492] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:53.401885 1168 tablet_bootstrap.cc:492] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.404429 1168 tablet_bootstrap.cc:654] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.406020 449 catalog_manager.cc:5526] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:53.406491 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]). Expected version 1 got 0
I20250115 14:51:53.407536 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d77ff794a83348c29391786666987db1
I20250115 14:51:53.408432 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d77ff794a83348c29391786666987db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.408596 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d77ff794a83348c29391786666987db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.408700 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d77ff794a83348c29391786666987db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:53.409250 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d77ff794a83348c29391786666987db1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d77ff794a83348c29391786666987db1 already in progress: creating tablet
W20250115 14:51:53.409960 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d77ff794a83348c29391786666987db1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d77ff794a83348c29391786666987db1 already in progress: creating tablet
I20250115 14:51:53.410288 1167 ts_tablet_manager.cc:1918] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.410431 1167 log.cc:1198] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/8d43cd66b18f4649894a629637037822
I20250115 14:51:53.410771 1167 ts_tablet_manager.cc:1939] T 8d43cd66b18f4649894a629637037822 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.410784 1168 tablet_bootstrap.cc:492] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.411041 1168 ts_tablet_manager.cc:1397] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20250115 14:51:53.411527 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d5ec136542e043e99db4296e025d725f" } new_table_name: "default.Table_86abf42208c1422c8cede2bf48d37c3a" modify_external_catalogs: true
I20250115 14:51:53.411845 1168 raft_consensus.cc:357] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.412043 1168 raft_consensus.cc:383] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.412111 1168 raft_consensus.cc:738] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.412446 1168 consensus_queue.cc:260] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.412951 1164 tablet_bootstrap.cc:654] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.413712 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b4259dfb8ee4b618c0bacc910addd0a with delete_type TABLET_DATA_DELETED (replaced by tablet 3a38a6bc32314396b497e7ae166a63f7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.413811 1167 tablet_replica.cc:331] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.413998 1167 raft_consensus.cc:2238] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.414033 1166 tablet_replica.cc:331] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:53.414162 1167 raft_consensus.cc:2267] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.414180 1166 raft_consensus.cc:2238] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.414347 1166 raft_consensus.cc:2267] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.414707 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 8d43cd66b18f4649894a629637037822 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]) successfully deleted
I20250115 14:51:53.415813 1168 ts_tablet_manager.cc:1428] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250115 14:51:53.416360 1162 tablet_bootstrap.cc:492] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.416893 1166 ts_tablet_manager.cc:1905] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.418074 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d4fb2d33f3c745beb4a4ba036e23a8e0
I20250115 14:51:53.418217 1168 tablet_bootstrap.cc:492] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.418470 450 catalog_manager.cc:6596] ReplaceTablet: tablet d4fb2d33f3c745beb4a4ba036e23a8e0 deleted and replaced by tablet 9f327ffa731c4c949b24a34dc54ae260
I20250115 14:51:53.419919 1342 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 14:51:53.420518 1167 ts_tablet_manager.cc:1905] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.420538 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f327ffa731c4c949b24a34dc54ae260) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.421473 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f327ffa731c4c949b24a34dc54ae260) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.421797 1168 tablet_bootstrap.cc:654] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.424086 1166 ts_tablet_manager.cc:1918] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.424260 1166 log.cc:1198] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/2b4259dfb8ee4b618c0bacc910addd0a
I20250115 14:51:53.424662 1166 ts_tablet_manager.cc:1939] T 2b4259dfb8ee4b618c0bacc910addd0a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:53.425683 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9126b20cb8474e8e9817459ddf0b9b0c" } new_table_name: "default.Table_24c05b9539414aadad44296c264be5aa" modify_external_catalogs: true
I20250115 14:51:53.425837 1162 ts_tablet_manager.cc:1397] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.004s
I20250115 14:51:53.419268 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f327ffa731c4c949b24a34dc54ae260) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:53.427412 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4fb2d33f3c745beb4a4ba036e23a8e0 already in progress: creating tablet
I20250115 14:51:53.427922 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 2b4259dfb8ee4b618c0bacc910addd0a (table default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]) successfully deleted
W20250115 14:51:53.428203 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4fb2d33f3c745beb4a4ba036e23a8e0 already in progress: creating tablet
I20250115 14:51:53.428251 1162 raft_consensus.cc:357] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.428483 1162 raft_consensus.cc:383] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.428562 1162 raft_consensus.cc:738] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.428833 1162 consensus_queue.cc:260] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.432577 1167 ts_tablet_manager.cc:1918] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.432700 1167 log.cc:1198] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ad67354f35ce4564b88dbdffcabf3564
I20250115 14:51:53.433048 1167 ts_tablet_manager.cc:1939] T ad67354f35ce4564b88dbdffcabf3564 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.433626 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet ad67354f35ce4564b88dbdffcabf3564 (table default.Table_bbf01a6c24b9449b8da3ee9428b7bd98 [id=23302b50545244ddb17c074740aacccd]) successfully deleted
I20250115 14:51:53.435118 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 5f508d8e23c14e48a438deb2d110b891
I20250115 14:51:53.435350 1167 tablet_replica.cc:331] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.435415 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3e3a408924cc4cb0b5a408089e63bb1e" } new_table_name: "default.Table_2898e18fe4e64a1a92ed3658b91dc502" modify_external_catalogs: true
I20250115 14:51:53.435552 1167 raft_consensus.cc:2238] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.435715 1167 raft_consensus.cc:2267] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.435863 1162 ts_tablet_manager.cc:1428] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250115 14:51:53.436636 1162 tablet_bootstrap.cc:492] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.439585 1162 tablet_bootstrap.cc:654] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.440740 1167 ts_tablet_manager.cc:1905] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.442732 1168 tablet_bootstrap.cc:492] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.442795 1221 fs_manager.cc:730] Time spent opening block manager: real 0.044s user 0.004s sys 0.002s
I20250115 14:51:53.442881 1168 ts_tablet_manager.cc:1397] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20250115 14:51:53.443583 1168 raft_consensus.cc:357] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.443780 1168 raft_consensus.cc:383] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.443866 1168 raft_consensus.cc:738] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.442989 1221 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data,/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
uuid: "1cc2e23c560f4096be7b14006bdfc525"
format_stamp: "Formatted at 2025-01-15 14:51:48 on dist-test-slave-pxr4"
I20250115 14:51:53.444101 1168 consensus_queue.cc:260] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.444206 1221 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 14:51:53.446054 1168 ts_tablet_manager.cc:1428] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:53.446369 448 master_service.cc:946] ReplaceTablet: received request to replace tablet d1fa787e364d4ca6a4b756a8f4beba3b from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.446503 1168 tablet_bootstrap.cc:492] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.449357 1168 tablet_bootstrap.cc:654] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.452688 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5d51e53b4ffa45bfbb9553d7a5870844
I20250115 14:51:53.453181 452 catalog_manager.cc:6596] ReplaceTablet: tablet 5d51e53b4ffa45bfbb9553d7a5870844 deleted and replaced by tablet b29ef998d8dc4d4b9bb0a824a850d824
I20250115 14:51:53.453363 1162 tablet_bootstrap.cc:492] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.453519 1162 ts_tablet_manager.cc:1397] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250115 14:51:53.454571 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d51e53b4ffa45bfbb9553d7a5870844 with delete_type TABLET_DATA_DELETED (replaced by tablet b29ef998d8dc4d4b9bb0a824a850d824) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.454661 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d51e53b4ffa45bfbb9553d7a5870844 with delete_type TABLET_DATA_DELETED (replaced by tablet b29ef998d8dc4d4b9bb0a824a850d824) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.454896 1165 tablet_replica.cc:331] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.454319 1162 raft_consensus.cc:357] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.455065 1162 raft_consensus.cc:383] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.455101 1165 raft_consensus.cc:2238] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.455210 1167 ts_tablet_manager.cc:1918] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.455278 1165 raft_consensus.cc:2267] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.455350 1167 log.cc:1198] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d77ff794a83348c29391786666987db1
I20250115 14:51:53.455569 1162 raft_consensus.cc:738] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.455950 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d51e53b4ffa45bfbb9553d7a5870844 with delete_type TABLET_DATA_DELETED (replaced by tablet b29ef998d8dc4d4b9bb0a824a850d824) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.455869 1162 consensus_queue.cc:260] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.456110 1165 ts_tablet_manager.cc:1905] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.456466 1162 ts_tablet_manager.cc:1428] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250115 14:51:53.457353 1162 tablet_bootstrap.cc:492] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:53.459475 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5d51e53b4ffa45bfbb9553d7a5870844 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d51e53b4ffa45bfbb9553d7a5870844 already in progress: creating tablet
I20250115 14:51:53.461948 1162 tablet_bootstrap.cc:654] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.465755 1165 ts_tablet_manager.cc:1918] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.465904 1165 log.cc:1198] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/5d51e53b4ffa45bfbb9553d7a5870844
I20250115 14:51:53.466225 1165 ts_tablet_manager.cc:1939] T 5d51e53b4ffa45bfbb9553d7a5870844 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.467272 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 5d51e53b4ffa45bfbb9553d7a5870844 (table default.Table_8003836a7a5c4f79a7bec455f2d8a325 [id=817d675e581a4abf848167ce48e46bd0]) successfully deleted
I20250115 14:51:53.467890 1167 ts_tablet_manager.cc:1939] T d77ff794a83348c29391786666987db1 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.469367 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d77ff794a83348c29391786666987db1 (table default.Table_26cfaab25925425bb14d5384a6ce1827 [id=6c9d98b8d6de494a9c6211022d1fc946]) successfully deleted
I20250115 14:51:53.469409 1167 tablet_replica.cc:331] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.469614 1167 raft_consensus.cc:2238] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.469805 1167 raft_consensus.cc:2267] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:53.469928 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4fb2d33f3c745beb4a4ba036e23a8e0 already in progress: creating tablet
I20250115 14:51:53.470319 1167 ts_tablet_manager.cc:1905] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.479990 1162 tablet_bootstrap.cc:492] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.480180 1162 ts_tablet_manager.cc:1397] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20250115 14:51:53.481031 1162 raft_consensus.cc:357] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.481261 1162 raft_consensus.cc:383] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.481346 1162 raft_consensus.cc:738] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.481591 1162 consensus_queue.cc:260] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.482100 1162 ts_tablet_manager.cc:1428] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:53.482399 1162 tablet_bootstrap.cc:492] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.482385 1164 tablet_bootstrap.cc:492] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:53.482523 1164 ts_tablet_manager.cc:1397] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.000s
I20250115 14:51:53.483276 1164 raft_consensus.cc:357] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.483807 1164 raft_consensus.cc:383] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.483902 1164 raft_consensus.cc:738] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.484031 1162 tablet_bootstrap.cc:654] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.484153 1164 consensus_queue.cc:260] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.487373 1167 ts_tablet_manager.cc:1918] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.487510 1167 log.cc:1198] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/5d51e53b4ffa45bfbb9553d7a5870844
I20250115 14:51:53.487854 1167 ts_tablet_manager.cc:1939] T 5d51e53b4ffa45bfbb9553d7a5870844 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.488131 1164 ts_tablet_manager.cc:1428] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250115 14:51:53.489586 1164 tablet_bootstrap.cc:492] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:53.492895 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c6c392518a464dab664e5f9321f1f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.492923 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 5d51e53b4ffa45bfbb9553d7a5870844 (table default.Table_8003836a7a5c4f79a7bec455f2d8a325 [id=817d675e581a4abf848167ce48e46bd0]) successfully deleted
I20250115 14:51:53.493145 1166 tablet_replica.cc:331] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:53.493317 1166 raft_consensus.cc:2238] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.493464 1166 raft_consensus.cc:2267] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.494215 1166 ts_tablet_manager.cc:1905] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.500430 1166 ts_tablet_manager.cc:1918] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.500634 1166 log.cc:1198] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/66c6c392518a464dab664e5f9321f1f6
I20250115 14:51:53.501047 1166 ts_tablet_manager.cc:1939] T 66c6c392518a464dab664e5f9321f1f6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:53.503775 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 66c6c392518a464dab664e5f9321f1f6 (table default.Table_7281d9e98ae843dc915b4932d3d7d50b [id=52b6d8da69f74139a394cafe66e48a35]) successfully deleted
I20250115 14:51:53.505362 1162 tablet_bootstrap.cc:492] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.505519 1162 ts_tablet_manager.cc:1397] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.005s
I20250115 14:51:53.506388 1162 raft_consensus.cc:357] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.506611 1162 raft_consensus.cc:383] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.506701 1162 raft_consensus.cc:738] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.506963 1162 consensus_queue.cc:260] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.511410 1162 ts_tablet_manager.cc:1428] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20250115 14:51:53.511406 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d77ff794a83348c29391786666987db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:53.511996 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d77ff794a83348c29391786666987db1 because the tablet was not found. No further retry: Not found: Tablet not found: d77ff794a83348c29391786666987db1
I20250115 14:51:53.514894 1162 tablet_bootstrap.cc:492] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.516768 1162 tablet_bootstrap.cc:654] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.518414 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 24751e606ad94675b650d67a3738481e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.518527 880 tablet_service.cc:1514] Processing DeleteTablet for tablet c56fe5b5b59b4017be3cb0ac877479b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.518683 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aed449a776493b8bdc7a31b47b62e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.518720 1167 tablet_replica.cc:331] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.518911 1167 raft_consensus.cc:2238] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.519096 1167 raft_consensus.cc:2267] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.519712 1167 ts_tablet_manager.cc:1905] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.520169 1164 tablet_bootstrap.cc:654] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.522518 1207 raft_consensus.cc:491] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.522667 1207 raft_consensus.cc:513] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.523411 1207 leader_election.cc:290] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.523394 1169 raft_consensus.cc:491] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.523547 1169 raft_consensus.cc:513] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.524185 1169 leader_election.cc:290] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:53.524189 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6205e20a1984ccabccc45b93f9c7fa8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.525178 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f960c88df14242c5ada1ecf9d8e68196" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.525377 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f960c88df14242c5ada1ecf9d8e68196" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:53.525626 912 raft_consensus.cc:2463] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:53.524190 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6205e20a1984ccabccc45b93f9c7fa8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:53.525929 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7e3ec000089a47a2930dec382192cd6f consensus state has no leader
I20250115 14:51:53.526010 1049 raft_consensus.cc:2463] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:53.526424 847 leader_election.cc:304] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:51:53.526450 984 leader_election.cc:343] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.526707 1207 raft_consensus.cc:2798] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:53.526886 1207 raft_consensus.cc:491] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250115 14:51:53.526852 848 leader_election.cc:343] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.526984 1207 raft_consensus.cc:3054] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:53.529985 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83097c5bfe364418a213236435edc40c consensus state has no leader
I20250115 14:51:53.530155 1207 raft_consensus.cc:513] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.530463 1167 ts_tablet_manager.cc:1918] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.530645 1167 log.cc:1198] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/24751e606ad94675b650d67a3738481e
I20250115 14:51:53.531292 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6205e20a1984ccabccc45b93f9c7fa8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:53.534696 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3ec4bc69bb7942fb9a2a2e5fe9b9bcbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:53.534941 983 leader_election.cc:304] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:53.535251 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ec4bc69bb7942fb9a2a2e5fe9b9bcbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.535403 1169 raft_consensus.cc:2798] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:53.535496 1169 raft_consensus.cc:491] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:53.535585 1169 raft_consensus.cc:3054] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:53.540478 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e0eba208f9be4a588cc81e4d99336eac consensus state has no leader
I20250115 14:51:53.540499 1169 raft_consensus.cc:513] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.541605 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f960c88df14242c5ada1ecf9d8e68196" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:53.542196 984 leader_election.cc:343] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:53.542414 848 leader_election.cc:343] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.543081 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6205e20a1984ccabccc45b93f9c7fa8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:53.543267 1049 raft_consensus.cc:3054] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.544023 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c368b4ef34f147b6984aee76adde3cf7" } new_table_name: "default.Table_253ac9c181914d0f891baf1f8722dbd0" modify_external_catalogs: true
I20250115 14:51:53.544503 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_eb6649c5d48f43e5959aa73d981a61cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.545038 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb6649c5d48f43e5959aa73d981a61cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.545843 1049 raft_consensus.cc:2463] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:53.546675 1169 leader_election.cc:290] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:53.547384 847 leader_election.cc:304] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.547837 1209 raft_consensus.cc:2798] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:53.547920 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f960c88df14242c5ada1ecf9d8e68196" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:53.548466 1167 ts_tablet_manager.cc:1939] T 24751e606ad94675b650d67a3738481e P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.549110 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 24751e606ad94675b650d67a3738481e (table default.Table_49eb379e719f4677b13269d15ef7efad [id=af78a0b6494d4fd68224992dec097051]) successfully deleted
I20250115 14:51:53.549634 1209 raft_consensus.cc:695] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:53.549908 1209 consensus_queue.cc:237] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.551373 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d1fa787e364d4ca6a4b756a8f4beba3b
I20250115 14:51:53.551843 448 catalog_manager.cc:6596] ReplaceTablet: tablet d1fa787e364d4ca6a4b756a8f4beba3b deleted and replaced by tablet 79aad920dd9744d9b87a8234c92232d8
I20250115 14:51:53.552934 912 raft_consensus.cc:3054] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.554148 1207 leader_election.cc:290] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.554311 1167 tablet_replica.cc:331] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.554493 1167 raft_consensus.cc:2238] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.554781 1167 raft_consensus.cc:2267] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.556522 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6674d45a98dc47b4b3c4abef46e52872" } new_table_name: "default.Table_39eb08eb474049f2a2fbe741ef730d68" modify_external_catalogs: true
I20250115 14:51:53.557384 912 raft_consensus.cc:2463] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:53.557849 983 leader_election.cc:304] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.557909 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d809e208790b4c1a911708b393be6a3f" } modify_external_catalogs: true
I20250115 14:51:53.558089 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d809e208790b4c1a911708b393be6a3f" } modify_external_catalogs: true
I20250115 14:51:53.558454 1169 raft_consensus.cc:2798] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:53.558616 1169 raft_consensus.cc:695] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:53.558907 1169 consensus_queue.cc:237] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.560091 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b with delete_type TABLET_DATA_DELETED (replaced by tablet 79aad920dd9744d9b87a8234c92232d8) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.560851 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b with delete_type TABLET_DATA_DELETED (replaced by tablet 79aad920dd9744d9b87a8234c92232d8) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:53.561187 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d1fa787e364d4ca6a4b756a8f4beba3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1fa787e364d4ca6a4b756a8f4beba3b already in progress: creating tablet
I20250115 14:51:53.561257 1167 ts_tablet_manager.cc:1905] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.564231 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_02593da1cd7245f38f85424d1a4baad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.564815 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02593da1cd7245f38f85424d1a4baad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.566455 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b with delete_type TABLET_DATA_DELETED (replaced by tablet 79aad920dd9744d9b87a8234c92232d8) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.566799 1207 raft_consensus.cc:491] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.566953 1207 raft_consensus.cc:513] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.567894 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a9fb932c5b4545a4609d568f0e5494" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.568229 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a9fb932c5b4545a4609d568f0e5494" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:53.568470 848 leader_election.cc:343] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.568535 1049 raft_consensus.cc:2463] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:53.568614 1207 leader_election.cc:290] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.569132 847 leader_election.cc:304] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.569598 1207 raft_consensus.cc:2798] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:53.569751 1207 raft_consensus.cc:491] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:53.569890 1207 raft_consensus.cc:3054] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:53.571275 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1839edd0d49b467d8bb31b3001ff89eb consensus state has no leader
I20250115 14:51:53.572057 455 master_service.cc:946] ReplaceTablet: received request to replace tablet b25dce523e724066b203e6111cad8111 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.572557 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2a2f7c950304406484083d56f0b3177e" } modify_external_catalogs: true
I20250115 14:51:53.572733 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_121cfa7da22441d581d821791e428785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:53.572847 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2a2f7c950304406484083d56f0b3177e" } modify_external_catalogs: true
W20250115 14:51:53.573276 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_121cfa7da22441d581d821791e428785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.573213 1207 raft_consensus.cc:513] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.574399 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a9fb932c5b4545a4609d568f0e5494" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:53.575021 848 leader_election.cc:343] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.575440 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2989f3ca246d42528088a1e17c79eb91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.575965 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2989f3ca246d42528088a1e17c79eb91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.576016 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a9fb932c5b4545a4609d568f0e5494" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:53.576228 1049 raft_consensus.cc:3054] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.577386 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e8aabd67059d46fe880f4b1262b8f9d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.577847 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8aabd67059d46fe880f4b1262b8f9d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:53.578372 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d1fa787e364d4ca6a4b756a8f4beba3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1fa787e364d4ca6a4b756a8f4beba3b already in progress: creating tablet
I20250115 14:51:53.579074 1049 raft_consensus.cc:2463] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:53.579636 847 leader_election.cc:304] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.581353 1207 leader_election.cc:290] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.583995 1357 raft_consensus.cc:2798] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:53.584151 1357 raft_consensus.cc:695] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:53.584656 1207 consensus_queue.cc:1035] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:53.584991 1357 consensus_queue.cc:237] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.586843 1168 tablet_bootstrap.cc:492] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.587005 1168 ts_tablet_manager.cc:1397] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.141s user 0.049s sys 0.067s
I20250115 14:51:53.587476 1164 tablet_bootstrap.cc:492] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:53.587638 1164 ts_tablet_manager.cc:1397] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.000s
I20250115 14:51:53.587700 1168 raft_consensus.cc:357] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.587924 1168 raft_consensus.cc:383] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.588337 1168 raft_consensus.cc:738] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.588436 1164 raft_consensus.cc:357] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.588645 1164 raft_consensus.cc:383] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.588747 1164 raft_consensus.cc:738] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.588625 1168 consensus_queue.cc:260] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.588979 1164 consensus_queue.cc:260] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.589262 1168 ts_tablet_manager.cc:1428] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:53.589789 1168 tablet_bootstrap.cc:492] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.590466 1164 ts_tablet_manager.cc:1428] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:53.590761 1164 tablet_bootstrap.cc:492] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:53.591302 1167 ts_tablet_manager.cc:1918] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.591488 1167 log.cc:1198] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/c56fe5b5b59b4017be3cb0ac877479b4
I20250115 14:51:53.591786 1168 tablet_bootstrap.cc:654] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.591974 1167 ts_tablet_manager.cc:1939] T c56fe5b5b59b4017be3cb0ac877479b4 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.593164 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet c56fe5b5b59b4017be3cb0ac877479b4 (table default.Table_740738b002f34e2dba618cd146ff8dcd [id=3e278eb470174b78a8cdaab8d9ab2464]) successfully deleted
I20250115 14:51:53.593240 1167 tablet_replica.cc:331] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.593523 1167 raft_consensus.cc:2238] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.593739 1167 raft_consensus.cc:2267] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.594344 1167 ts_tablet_manager.cc:1905] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.594535 1169 raft_consensus.cc:491] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.594691 1169 raft_consensus.cc:513] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.595747 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22edaa6e70d942ba98777973476f449d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:53.596344 984 leader_election.cc:343] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.597291 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22edaa6e70d942ba98777973476f449d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:53.597503 912 raft_consensus.cc:2463] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:53.597909 983 leader_election.cc:304] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.598196 1225 raft_consensus.cc:2798] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:53.598289 1225 raft_consensus.cc:491] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:53.598359 1225 raft_consensus.cc:3054] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.600970 1168 tablet_bootstrap.cc:492] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.600948 1225 raft_consensus.cc:513] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.601094 1168 ts_tablet_manager.cc:1397] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20250115 14:51:53.601881 1225 leader_election.cc:290] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:53.602110 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 22edaa6e70d942ba98777973476f449d (table default.Table_1b3a658f6eda417cb956eba9cdc5a2af [id=d7ba4cda40964e1d839efe9e0f1615c0]). Expected version 3 got 0
I20250115 14:51:53.602768 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22edaa6e70d942ba98777973476f449d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:53.603399 984 leader_election.cc:343] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.603837 1169 leader_election.cc:290] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:53.604403 1168 raft_consensus.cc:357] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.604614 1168 raft_consensus.cc:383] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.604542 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22edaa6e70d942ba98777973476f449d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:53.605304 1168 raft_consensus.cc:738] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.605396 912 raft_consensus.cc:3054] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.605301 452 catalog_manager.cc:5526] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:53.605576 1168 consensus_queue.cc:260] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.607862 886 tablet_service.cc:1467] Processing CreateTablet for tablet 50ca710cea154be5b0f57ace4af4168b (DEFAULT_TABLE table=default.Table_d809e208790b4c1a911708b393be6a3f [id=d45a9a753a874c41a842f25450ab10d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.608238 1168 ts_tablet_manager.cc:1428] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:51:53.608217 885 tablet_service.cc:1467] Processing CreateTablet for tablet f50edca843e54f4b8e803d4d180b736d (DEFAULT_TABLE table=default.Table_490c22b7496a4d7db6bc6955e246363b [id=93b72d2ab0bc4ca2bf675a110a5bf4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.608345 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 50ca710cea154be5b0f57ace4af4168b (DEFAULT_TABLE table=default.Table_d809e208790b4c1a911708b393be6a3f [id=d45a9a753a874c41a842f25450ab10d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.608541 1162 tablet_bootstrap.cc:492] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.608767 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ca710cea154be5b0f57ace4af4168b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.608903 1168 tablet_bootstrap.cc:492] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.608867 890 tablet_service.cc:1467] Processing CreateTablet for tablet b9ce445d334842c8a6bacad9f693feda (DEFAULT_TABLE table=default.Table_ff3a0751d563485db14fc5a44116f2ac [id=4334f62c5f0a43119d21690c19315404]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.608878 1162 ts_tablet_manager.cc:1397] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.011s
I20250115 14:51:53.609483 880 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9e2c053d624b6ba71b031a591475a4 (DEFAULT_TABLE table=default.Table_f88c333bf687464cb16fb61569488e70 [id=2c2d7017c98f49d09e57327f21412e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.609836 1162 raft_consensus.cc:357] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.610069 1162 raft_consensus.cc:383] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.610144 1162 raft_consensus.cc:738] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.610334 888 tablet_service.cc:1467] Processing CreateTablet for tablet d7309ecb11d844848638c042c9bbe5f1 (DEFAULT_TABLE table=default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.610411 1162 consensus_queue.cc:260] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.610875 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f50edca843e54f4b8e803d4d180b736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.610924 1162 ts_tablet_manager.cc:1428] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:53.612398 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9e2c053d624b6ba71b031a591475a4 (DEFAULT_TABLE table=default.Table_f88c333bf687464cb16fb61569488e70 [id=2c2d7017c98f49d09e57327f21412e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.612833 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9e2c053d624b6ba71b031a591475a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.612855 1162 tablet_bootstrap.cc:492] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.613371 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ce445d334842c8a6bacad9f693feda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.614509 1162 tablet_bootstrap.cc:654] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.615588 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7309ecb11d844848638c042c9bbe5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.616413 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9e2c053d624b6ba71b031a591475a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.616603 1022 tablet_service.cc:1467] Processing CreateTablet for tablet b9ce445d334842c8a6bacad9f693feda (DEFAULT_TABLE table=default.Table_ff3a0751d563485db14fc5a44116f2ac [id=4334f62c5f0a43119d21690c19315404]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.616976 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ce445d334842c8a6bacad9f693feda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.617928 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ca710cea154be5b0f57ace4af4168b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.620631 1021 tablet_service.cc:1467] Processing CreateTablet for tablet f50edca843e54f4b8e803d4d180b736d (DEFAULT_TABLE table=default.Table_490c22b7496a4d7db6bc6955e246363b [id=93b72d2ab0bc4ca2bf675a110a5bf4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.621019 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f50edca843e54f4b8e803d4d180b736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.624648 1027 tablet_service.cc:1467] Processing CreateTablet for tablet d7309ecb11d844848638c042c9bbe5f1 (DEFAULT_TABLE table=default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.625048 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7309ecb11d844848638c042c9bbe5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.628979 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet c322301320ba425a94d7321c08f7f9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.629207 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d43cd66b18f4649894a629637037822 with delete_type TABLET_DATA_DELETED (replaced by tablet bb792ca1b3f54def92f504ec8682d2c7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.629493 1165 tablet_replica.cc:331] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.629698 1165 raft_consensus.cc:2238] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.629871 1165 raft_consensus.cc:2267] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:53.631130 848 consensus_peers.cc:487] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:53.631623 1165 ts_tablet_manager.cc:1905] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.637980 1168 tablet_bootstrap.cc:654] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.640159 1165 ts_tablet_manager.cc:1918] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.640928 1167 ts_tablet_manager.cc:1918] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.641079 1167 log.cc:1198] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/69aed449a776493b8bdc7a31b47b62e8
I20250115 14:51:53.641404 1167 ts_tablet_manager.cc:1939] T 69aed449a776493b8bdc7a31b47b62e8 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.641897 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 69aed449a776493b8bdc7a31b47b62e8 (table default.Table_68b340d318b14c078d634b22c471d979 [id=e2d228ef5e814fd9bf50e2ed7e89be4d]) successfully deleted
W20250115 14:51:53.642200 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d1fa787e364d4ca6a4b756a8f4beba3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1fa787e364d4ca6a4b756a8f4beba3b already in progress: creating tablet
I20250115 14:51:53.642915 1168 tablet_bootstrap.cc:492] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.643078 1168 ts_tablet_manager.cc:1397] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250115 14:51:53.644403 1168 raft_consensus.cc:357] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.644632 1168 raft_consensus.cc:383] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.644707 1168 raft_consensus.cc:738] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.644953 912 raft_consensus.cc:2463] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:53.646010 1168 consensus_queue.cc:260] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.651882 1168 ts_tablet_manager.cc:1428] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250115 14:51:53.654251 1168 tablet_bootstrap.cc:492] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.661906 1168 tablet_bootstrap.cc:654] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.645728 1162 tablet_bootstrap.cc:492] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.671759 1162 ts_tablet_manager.cc:1397] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.003s
I20250115 14:51:53.672727 1162 raft_consensus.cc:357] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.672956 1162 raft_consensus.cc:383] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.673060 1162 raft_consensus.cc:738] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.673285 1162 consensus_queue.cc:260] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.673786 1162 ts_tablet_manager.cc:1428] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:53.674096 1162 tablet_bootstrap.cc:492] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.675774 1162 tablet_bootstrap.cc:654] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.646111 983 leader_election.cc:304] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.676620 1225 raft_consensus.cc:2798] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:53.676770 1225 raft_consensus.cc:695] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:53.677112 1165 log.cc:1198] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/c322301320ba425a94d7321c08f7f9e6
I20250115 14:51:53.677083 1225 consensus_queue.cc:237] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.677533 1165 ts_tablet_manager.cc:1939] T c322301320ba425a94d7321c08f7f9e6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.678431 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet c322301320ba425a94d7321c08f7f9e6 (table default.Table_dd152d863f394374a58bf845d3d259e2 [id=d5216eacff2b46ad96642b0ef7fd6403]) successfully deleted
I20250115 14:51:53.678534 1168 tablet_bootstrap.cc:492] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.679634 1165 tablet_replica.cc:331] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.679661 1168 ts_tablet_manager.cc:1397] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.005s
I20250115 14:51:53.679842 1165 raft_consensus.cc:2238] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:53.680066 1165 pending_rounds.cc:62] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:53.680150 1165 pending_rounds.cc:69] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558313778974720 op_type: NO_OP noop_request { }
I20250115 14:51:53.680263 1165 raft_consensus.cc:2883] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:53.680400 1165 raft_consensus.cc:2267] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.680449 1168 raft_consensus.cc:357] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.680637 1168 raft_consensus.cc:383] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.680733 1168 raft_consensus.cc:738] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.680971 1168 consensus_queue.cc:260] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.681492 1168 ts_tablet_manager.cc:1428] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:53.681788 1168 tablet_bootstrap.cc:492] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.683548 1168 tablet_bootstrap.cc:654] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.685297 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 745153c106844076bb020f9ed026c97a from {username='slave'} at 127.0.0.1:47464
W20250115 14:51:53.680991 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:45653, user_credentials={real_user=slave}} blocked reactor thread for 83413us
I20250115 14:51:53.689924 749 tablet_service.cc:1467] Processing CreateTablet for tablet 50ca710cea154be5b0f57ace4af4168b (DEFAULT_TABLE table=default.Table_d809e208790b4c1a911708b393be6a3f [id=d45a9a753a874c41a842f25450ab10d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.690402 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ca710cea154be5b0f57ace4af4168b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.690757 754 tablet_service.cc:1467] Processing CreateTablet for tablet b9ce445d334842c8a6bacad9f693feda (DEFAULT_TABLE table=default.Table_ff3a0751d563485db14fc5a44116f2ac [id=4334f62c5f0a43119d21690c19315404]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.691118 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ce445d334842c8a6bacad9f693feda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.691660 1165 ts_tablet_manager.cc:1905] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.696578 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9e2c053d624b6ba71b031a591475a4 (DEFAULT_TABLE table=default.Table_f88c333bf687464cb16fb61569488e70 [id=2c2d7017c98f49d09e57327f21412e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.696964 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a9e2c053d624b6ba71b031a591475a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.697628 753 tablet_service.cc:1467] Processing CreateTablet for tablet d7309ecb11d844848638c042c9bbe5f1 (DEFAULT_TABLE table=default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.698007 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7309ecb11d844848638c042c9bbe5f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.700392 747 tablet_service.cc:1467] Processing CreateTablet for tablet f50edca843e54f4b8e803d4d180b736d (DEFAULT_TABLE table=default.Table_490c22b7496a4d7db6bc6955e246363b [id=93b72d2ab0bc4ca2bf675a110a5bf4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:53.700724 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f50edca843e54f4b8e803d4d180b736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:53.702144 1164 tablet_bootstrap.cc:654] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.709144 1168 tablet_bootstrap.cc:492] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.709328 1168 ts_tablet_manager.cc:1397] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20250115 14:51:53.710363 1168 raft_consensus.cc:357] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.710582 1168 raft_consensus.cc:383] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.710654 1168 raft_consensus.cc:738] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.710924 1168 consensus_queue.cc:260] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.711417 1168 ts_tablet_manager.cc:1428] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:53.712635 1165 ts_tablet_manager.cc:1918] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:53.714016 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d2529350d9264a1793c605ea9d5df9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fad6f9f7f853418e85353edfd38cc80e) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.714272 1168 tablet_bootstrap.cc:492] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.714274 1166 tablet_replica.cc:331] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:53.714442 1166 raft_consensus.cc:2238] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.712767 1165 log.cc:1198] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/8d43cd66b18f4649894a629637037822
I20250115 14:51:53.714635 1166 raft_consensus.cc:2267] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.714846 1165 ts_tablet_manager.cc:1939] T 8d43cd66b18f4649894a629637037822 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.717603 1168 tablet_bootstrap.cc:654] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.718787 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 8d43cd66b18f4649894a629637037822 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]) successfully deleted
I20250115 14:51:53.719936 1049 raft_consensus.cc:1270] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:53.719998 1166 ts_tablet_manager.cc:1905] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.720834 1209 consensus_queue.cc:1035] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:51:53.723742 848 consensus_peers.cc:487] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:53.727526 1166 ts_tablet_manager.cc:1918] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.727687 1166 log.cc:1198] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d2529350d9264a1793c605ea9d5df9c6
I20250115 14:51:53.728032 1166 ts_tablet_manager.cc:1939] T d2529350d9264a1793c605ea9d5df9c6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:53.729378 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c68a808d4c4744fe92a97753d517180a" } modify_external_catalogs: true
I20250115 14:51:53.729637 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c68a808d4c4744fe92a97753d517180a" } modify_external_catalogs: true
I20250115 14:51:53.736522 1162 tablet_bootstrap.cc:492] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.737068 1162 ts_tablet_manager.cc:1397] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.003s
I20250115 14:51:53.737946 1162 raft_consensus.cc:357] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.739323 1164 tablet_bootstrap.cc:492] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:53.739539 1164 ts_tablet_manager.cc:1397] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.149s user 0.004s sys 0.000s
I20250115 14:51:53.740387 1164 raft_consensus.cc:357] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.740657 1164 raft_consensus.cc:383] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.740787 1164 raft_consensus.cc:738] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.741055 1164 consensus_queue.cc:260] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.742386 1164 ts_tablet_manager.cc:1428] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250115 14:51:53.742714 1164 tablet_bootstrap.cc:492] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:53.742889 1225 raft_consensus.cc:491] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.745007 1162 raft_consensus.cc:383] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.745107 1162 raft_consensus.cc:738] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.745072 1225 raft_consensus.cc:513] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.745354 1162 consensus_queue.cc:260] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.745662 1225 leader_election.cc:290] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:53.745899 1162 ts_tablet_manager.cc:1428] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250115 14:51:53.746217 1162 tablet_bootstrap.cc:492] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.746668 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0eba208f9be4a588cc81e4d99336eac" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:53.746920 912 raft_consensus.cc:2463] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:53.747828 1162 tablet_bootstrap.cc:654] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.747956 983 leader_election.cc:304] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:53.748354 1169 raft_consensus.cc:2798] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:53.748476 1169 raft_consensus.cc:491] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:53.748562 1169 raft_consensus.cc:3054] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.749001 1164 tablet_bootstrap.cc:654] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.750893 1168 tablet_bootstrap.cc:492] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.751029 1168 ts_tablet_manager.cc:1397] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.010s
I20250115 14:51:53.751118 1169 raft_consensus.cc:513] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.752391 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d2529350d9264a1793c605ea9d5df9c6 (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]) successfully deleted
I20250115 14:51:53.752342 1168 raft_consensus.cc:357] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.752530 1168 raft_consensus.cc:383] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.752622 1168 raft_consensus.cc:738] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.752804 1207 raft_consensus.cc:491] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.752877 1168 consensus_queue.cc:260] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.752990 1207 raft_consensus.cc:513] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.753414 1168 ts_tablet_manager.cc:1428] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:53.753459 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0eba208f9be4a588cc81e4d99336eac" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.753618 1207 leader_election.cc:290] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.753873 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0eba208f9be4a588cc81e4d99336eac" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:53.754468 984 leader_election.cc:343] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.754633 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4161fce81834cd2ba4ffb789f6a5ba3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.754746 1168 tablet_bootstrap.cc:492] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:51:53.754791 984 leader_election.cc:343] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.755592 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0eba208f9be4a588cc81e4d99336eac" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
W20250115 14:51:53.755635 848 leader_election.cc:343] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.755795 912 raft_consensus.cc:3054] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.756356 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4161fce81834cd2ba4ffb789f6a5ba3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:53.756533 1168 tablet_bootstrap.cc:654] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.756570 1049 raft_consensus.cc:2463] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:53.756982 847 leader_election.cc:304] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.757422 1209 raft_consensus.cc:2798] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:53.757566 1209 raft_consensus.cc:491] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:53.757645 1209 raft_consensus.cc:3054] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.758579 912 raft_consensus.cc:2463] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:53.759093 983 leader_election.cc:304] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.760520 1225 raft_consensus.cc:2798] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:53.760648 1225 raft_consensus.cc:695] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:53.760618 1209 raft_consensus.cc:513] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.760896 1225 consensus_queue.cc:237] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.761231 1209 leader_election.cc:290] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.761905 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4161fce81834cd2ba4ffb789f6a5ba3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:53.762149 1049 raft_consensus.cc:3054] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:53.763170 1169 leader_election.cc:290] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:53.763370 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4161fce81834cd2ba4ffb789f6a5ba3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:53.763911 848 leader_election.cc:343] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.766533 1049 raft_consensus.cc:2463] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:53.766965 847 leader_election.cc:304] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.767393 1209 raft_consensus.cc:2798] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:53.767530 1209 raft_consensus.cc:695] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:53.767843 1209 consensus_queue.cc:237] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.776371 1221 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 14:51:53.777518 1221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 14:51:53.777840 1221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 14:51:53.785802 1168 tablet_bootstrap.cc:492] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.786010 1168 ts_tablet_manager.cc:1397] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.004s
I20250115 14:51:53.786873 1168 raft_consensus.cc:357] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.787123 1168 raft_consensus.cc:383] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.787195 1168 raft_consensus.cc:738] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.787452 1168 consensus_queue.cc:260] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.788169 1168 ts_tablet_manager.cc:1428] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:53.789494 1375 tablet.cc:1721] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:53.799726 1168 tablet_bootstrap.cc:492] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.801604 1207 raft_consensus.cc:491] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.801782 1207 raft_consensus.cc:513] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.803668 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f312f1e3a33a404ca7284c713859f156" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.803912 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f312f1e3a33a404ca7284c713859f156" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:53.808377 1207 leader_election.cc:290] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:51:53.808544 847 leader_election.cc:343] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:53.808971 848 leader_election.cc:343] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:53.809002 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86f0144904ae4008945ccf325039dac1 consensus state has no leader
I20250115 14:51:53.809154 848 leader_election.cc:304] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.809605 1207 raft_consensus.cc:2743] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:53.810308 1374 tablet.cc:1721] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:53.812479 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_490c22b7496a4d7db6bc6955e246363b" } modify_external_catalogs: true
I20250115 14:51:53.812700 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_490c22b7496a4d7db6bc6955e246363b" } modify_external_catalogs: true
I20250115 14:51:53.813284 1168 tablet_bootstrap.cc:654] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.814963 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8003836a7a5c4f79a7bec455f2d8a325" } modify_external_catalogs: true
I20250115 14:51:53.815146 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8003836a7a5c4f79a7bec455f2d8a325" } modify_external_catalogs: true
I20250115 14:51:53.816517 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_17b896568dc34b1aaf2b3e197d03fdc4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.817106 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17b896568dc34b1aaf2b3e197d03fdc4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.818354 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 259ac764ca844df9b6c0ce1688a188da from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.819406 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 07a63a0a8bb44fe9be0740fc777e610c
I20250115 14:51:53.819969 454 master_service.cc:946] ReplaceTablet: received request to replace tablet d243daba84ff4befa0c4e7bac5fe7476 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.820567 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c75173371e554bdb8d99dcf3cea7c244"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:53.820986 747 tablet_service.cc:1514] Processing DeleteTablet for tablet d2529350d9264a1793c605ea9d5df9c6 with delete_type TABLET_DATA_DELETED (replaced by tablet fad6f9f7f853418e85353edfd38cc80e) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:53.821098 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c75173371e554bdb8d99dcf3cea7c244 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.822213 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 50ca710cea154be5b0f57ace4af4168b
I20250115 14:51:53.822484 1209 raft_consensus.cc:491] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.822639 1209 raft_consensus.cc:513] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.822847 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_42f23bc0b4dd4745923028c81cf9ac37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.823341 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42f23bc0b4dd4745923028c81cf9ac37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.823637 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd72a0d586d4a2c817f6a39daa07063" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:53.824190 848 leader_election.cc:343] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.825544 1168 tablet_bootstrap.cc:492] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.825688 1168 ts_tablet_manager.cc:1397] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.000s
I20250115 14:51:53.826023 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a63a0a8bb44fe9be0740fc777e610c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.826443 1168 raft_consensus.cc:357] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.826647 1168 raft_consensus.cc:383] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.826740 1168 raft_consensus.cc:738] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.827061 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ca710cea154be5b0f57ace4af4168b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.826998 1168 consensus_queue.cc:260] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.827565 1168 ts_tablet_manager.cc:1428] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:53.827831 1168 tablet_bootstrap.cc:492] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.830602 1168 tablet_bootstrap.cc:654] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.832576 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a63a0a8bb44fe9be0740fc777e610c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.832836 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ca710cea154be5b0f57ace4af4168b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.833407 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b25dce523e724066b203e6111cad8111
I20250115 14:51:53.833827 455 catalog_manager.cc:6596] ReplaceTablet: tablet b25dce523e724066b203e6111cad8111 deleted and replaced by tablet 0de6f6f014074c3d8775cb5e4121ccb0
I20250115 14:51:53.834220 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c3fec71fedac4b4c9864b18ea6af10b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.834728 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3fec71fedac4b4c9864b18ea6af10b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.836608 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b25dce523e724066b203e6111cad8111 with delete_type TABLET_DATA_DELETED (replaced by tablet 0de6f6f014074c3d8775cb5e4121ccb0) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:53.836745 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 07a63a0a8bb44fe9be0740fc777e610c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a63a0a8bb44fe9be0740fc777e610c already in progress: creating tablet
W20250115 14:51:53.837019 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 50ca710cea154be5b0f57ace4af4168b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ca710cea154be5b0f57ace4af4168b already in progress: creating tablet
W20250115 14:51:53.837263 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b25dce523e724066b203e6111cad8111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b25dce523e724066b203e6111cad8111 already in progress: creating tablet
I20250115 14:51:53.838477 1209 leader_election.cc:290] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.845017 1209 raft_consensus.cc:491] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.845165 1209 raft_consensus.cc:513] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.846117 1164 tablet_bootstrap.cc:492] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:53.846110 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d95c9359e54f1a88a07d5c5cce7fa1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.846637 1164 ts_tablet_manager.cc:1397] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.104s user 0.000s sys 0.017s
W20250115 14:51:53.847067 848 leader_election.cc:343] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.847312 1209 leader_election.cc:290] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.848639 1374 tablet.cc:1721] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:53.849480 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a63a0a8bb44fe9be0740fc777e610c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.849545 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d95c9359e54f1a88a07d5c5cce7fa1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:53.849757 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ca710cea154be5b0f57ace4af4168b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.850006 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet b25dce523e724066b203e6111cad8111 with delete_type TABLET_DATA_DELETED (replaced by tablet 0de6f6f014074c3d8775cb5e4121ccb0) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.850447 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd72a0d586d4a2c817f6a39daa07063" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:53.847422 1164 raft_consensus.cc:357] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.852139 1164 raft_consensus.cc:383] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.852253 1164 raft_consensus.cc:738] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.852600 1164 consensus_queue.cc:260] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.854075 1162 tablet_bootstrap.cc:492] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.854543 1162 ts_tablet_manager.cc:1397] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.014s
I20250115 14:51:53.855770 1162 raft_consensus.cc:357] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:53.857267 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 07a63a0a8bb44fe9be0740fc777e610c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a63a0a8bb44fe9be0740fc777e610c already in progress: creating tablet
W20250115 14:51:53.857522 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 50ca710cea154be5b0f57ace4af4168b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ca710cea154be5b0f57ace4af4168b already in progress: creating tablet
W20250115 14:51:53.857720 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b25dce523e724066b203e6111cad8111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b25dce523e724066b203e6111cad8111 already in progress: creating tablet
I20250115 14:51:53.860059 1164 ts_tablet_manager.cc:1428] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250115 14:51:53.860472 1164 tablet_bootstrap.cc:492] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:51:53.850220 847 leader_election.cc:343] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.864496 847 leader_election.cc:304] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:53.865088 847 leader_election.cc:343] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.865195 847 leader_election.cc:304] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.865686 1207 raft_consensus.cc:2743] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:53.865849 1207 raft_consensus.cc:2743] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:53.871181 1357 raft_consensus.cc:491] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.871395 1357 raft_consensus.cc:513] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.873246 1357 leader_election.cc:290] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:51:53.876870 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7a9e2c053d624b6ba71b031a591475a4 (table default.Table_f88c333bf687464cb16fb61569488e70 [id=2c2d7017c98f49d09e57327f21412e91], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7a9e2c053d624b6ba71b031a591475a4 consensus state has no leader
W20250115 14:51:53.877403 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d2529350d9264a1793c605ea9d5df9c6 because the tablet was not found. No further retry: Not found: Tablet not found: d2529350d9264a1793c605ea9d5df9c6
W20250115 14:51:53.877631 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 07a63a0a8bb44fe9be0740fc777e610c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a63a0a8bb44fe9be0740fc777e610c already in progress: creating tablet
W20250115 14:51:53.877835 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 50ca710cea154be5b0f57ace4af4168b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ca710cea154be5b0f57ace4af4168b already in progress: creating tablet
I20250115 14:51:53.882339 1168 tablet_bootstrap.cc:492] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.882540 1168 ts_tablet_manager.cc:1397] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.055s user 0.016s sys 0.033s
I20250115 14:51:53.883275 1168 raft_consensus.cc:357] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.884140 1168 raft_consensus.cc:383] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.884258 1168 raft_consensus.cc:738] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.884575 1168 consensus_queue.cc:260] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.870666 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b25dce523e724066b203e6111cad8111 with delete_type TABLET_DATA_DELETED (replaced by tablet 0de6f6f014074c3d8775cb5e4121ccb0) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.872385 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad6f9f7f853418e85353edfd38cc80e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.889758 1168 ts_tablet_manager.cc:1428] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20250115 14:51:53.890647 848 leader_election.cc:343] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.891163 1168 tablet_bootstrap.cc:492] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.893071 1168 tablet_bootstrap.cc:654] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.902273 1168 tablet_bootstrap.cc:492] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.902407 1168 ts_tablet_manager.cc:1397] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20250115 14:51:53.903402 1168 raft_consensus.cc:357] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.903618 1168 raft_consensus.cc:383] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.903695 1168 raft_consensus.cc:738] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.903932 1168 consensus_queue.cc:260] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.905123 1162 raft_consensus.cc:383] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.905848 1168 ts_tablet_manager.cc:1428] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:53.906275 1168 tablet_bootstrap.cc:492] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.873085 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad6f9f7f853418e85353edfd38cc80e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:53.904594 1225 raft_consensus.cc:491] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.904798 1169 raft_consensus.cc:491] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.908689 1169 raft_consensus.cc:513] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:53.908973 847 leader_election.cc:343] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.909087 847 leader_election.cc:304] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.909459 1357 raft_consensus.cc:2743] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:53.858354 1394 tablet.cc:1721] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:53.910229 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24751e606ad94675b650d67a3738481e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.910336 1162 raft_consensus.cc:738] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.910908 1162 consensus_queue.cc:260] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.910931 1225 raft_consensus.cc:513] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:53.911396 983 leader_election.cc:343] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 24751e606ad94675b650d67a3738481e
I20250115 14:51:53.911515 1162 ts_tablet_manager.cc:1428] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.056s user 0.000s sys 0.001s
I20250115 14:51:53.911859 1162 tablet_bootstrap.cc:492] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:53.913344 984 leader_election.cc:343] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.913460 984 leader_election.cc:304] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.913902 1168 tablet_bootstrap.cc:654] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.914181 1231 raft_consensus.cc:2743] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:53.914530 1169 leader_election.cc:290] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:53.914871 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b25dce523e724066b203e6111cad8111 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b25dce523e724066b203e6111cad8111 already in progress: creating tablet
I20250115 14:51:53.915206 1225 leader_election.cc:290] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:53.917191 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad67354f35ce4564b88dbdffcabf3564" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.919018 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f50edca843e54f4b8e803d4d180b736d
I20250115 14:51:53.920833 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet f50edca843e54f4b8e803d4d180b736d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.920958 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_475c872b794349c397ee73627f500616" } new_table_name: "default.Table_9984f5b933e4431ba6201377446f3026" modify_external_catalogs: true
W20250115 14:51:53.921562 983 leader_election.cc:343] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: ad67354f35ce4564b88dbdffcabf3564
I20250115 14:51:53.922211 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b29ef998d8dc4d4b9bb0a824a850d824
W20250115 14:51:53.922178 984 leader_election.cc:343] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:53.922289 984 leader_election.cc:304] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:53.922484 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3ec4bc69bb7942fb9a2a2e5fe9b9bcbd" } new_table_name: "default.Table_da0cb47ea83b436d9a84f797bc9b02d0" modify_external_catalogs: true
I20250115 14:51:53.922667 1225 raft_consensus.cc:2743] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:53.922966 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f50edca843e54f4b8e803d4d180b736d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f50edca843e54f4b8e803d4d180b736d already in progress: creating tablet
I20250115 14:51:53.923923 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c7fd75443d3a4d108674df939f2c67f7" } new_table_name: "default.Table_86054daadda14e4b834b785a68f5e02e" modify_external_catalogs: true
I20250115 14:51:53.924038 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f50edca843e54f4b8e803d4d180b736d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.925117 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 259ac764ca844df9b6c0ce1688a188da
I20250115 14:51:53.925527 451 catalog_manager.cc:6596] ReplaceTablet: tablet 259ac764ca844df9b6c0ce1688a188da deleted and replaced by tablet bc8337d181d44ac7bb82551a677b6831
I20250115 14:51:53.925864 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_92f683675cf0474db415bd700c1c4178"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.926374 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92f683675cf0474db415bd700c1c4178 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.927515 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 259ac764ca844df9b6c0ce1688a188da with delete_type TABLET_DATA_DELETED (replaced by tablet bc8337d181d44ac7bb82551a677b6831) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.927678 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4b96cde69a5b411f97fe0f70ae0264c0" } new_table_name: "default.Table_27cd4d793c8f4e77aa531c06dfa74a2d" modify_external_catalogs: true
W20250115 14:51:53.931033 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 259ac764ca844df9b6c0ce1688a188da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 259ac764ca844df9b6c0ce1688a188da already in progress: creating tablet
I20250115 14:51:53.932263 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 259ac764ca844df9b6c0ce1688a188da with delete_type TABLET_DATA_DELETED (replaced by tablet bc8337d181d44ac7bb82551a677b6831) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:53.932319 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f50edca843e54f4b8e803d4d180b736d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f50edca843e54f4b8e803d4d180b736d already in progress: creating tablet
I20250115 14:51:53.933133 1164 tablet_bootstrap.cc:654] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:53.933265 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 259ac764ca844df9b6c0ce1688a188da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 259ac764ca844df9b6c0ce1688a188da already in progress: creating tablet
I20250115 14:51:53.934458 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f50edca843e54f4b8e803d4d180b736d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.935317 747 tablet_service.cc:1514] Processing DeleteTablet for tablet e115bdfa1d69421890122f9afdc23b29 with delete_type TABLET_DATA_DELETED (replaced by tablet f312f1e3a33a404ca7284c713859f156) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.935580 1166 tablet_replica.cc:331] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:53.935784 1166 raft_consensus.cc:2238] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.935942 1166 raft_consensus.cc:2267] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.939642 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 259ac764ca844df9b6c0ce1688a188da with delete_type TABLET_DATA_DELETED (replaced by tablet bc8337d181d44ac7bb82551a677b6831) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.940284 1166 ts_tablet_manager.cc:1905] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:53.941241 1162 tablet_bootstrap.cc:654] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.942837 1164 tablet_bootstrap.cc:492] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:53.942973 1164 ts_tablet_manager.cc:1397] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.083s user 0.003s sys 0.000s
I20250115 14:51:53.943725 1164 raft_consensus.cc:357] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.943961 1164 raft_consensus.cc:383] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.944046 1164 raft_consensus.cc:738] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.947218 1166 ts_tablet_manager.cc:1918] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.947387 1166 log.cc:1198] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e115bdfa1d69421890122f9afdc23b29
I20250115 14:51:53.947767 1166 ts_tablet_manager.cc:1939] T e115bdfa1d69421890122f9afdc23b29 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:53.948443 1164 consensus_queue.cc:260] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.949853 1164 ts_tablet_manager.cc:1428] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250115 14:51:53.950330 1164 tablet_bootstrap.cc:492] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:51:53.952090 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f50edca843e54f4b8e803d4d180b736d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f50edca843e54f4b8e803d4d180b736d already in progress: creating tablet
I20250115 14:51:53.952536 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e115bdfa1d69421890122f9afdc23b29 (table default.Table_2752e5f3a4ca4c41a614738b9354d962 [id=1c779c8a53f54707abe1d619b30a912e]) successfully deleted
W20250115 14:51:53.959204 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31d95c9359e54f1a88a07d5c5cce7fa1 consensus state has no leader
I20250115 14:51:53.959554 449 master_service.cc:946] ReplaceTablet: received request to replace tablet fe0c1c829aff438a94bc41609bdb7841 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.959658 446 catalog_manager.cc:5526] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:53.960098 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 22edaa6e70d942ba98777973476f449d (table default.Table_1b3a658f6eda417cb956eba9cdc5a2af [id=d7ba4cda40964e1d839efe9e0f1615c0]). Expected version 3 got 0
I20250115 14:51:53.960603 446 catalog_manager.cc:5526] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:53.961014 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet f960c88df14242c5ada1ecf9d8e68196 (table default.Table_56a365aae805419683d1f4dfd97951cc [id=24d4e203bce64fcdbd5b6fa30a99e6b2]). Expected version 1 got 0
I20250115 14:51:53.961243 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]). Expected version 2 got 0
I20250115 14:51:53.961397 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet e0eba208f9be4a588cc81e4d99336eac (table default.Table_2898e18fe4e64a1a92ed3658b91dc502 [id=6d0e848982e245e1870c661f7dc35138]). Expected version 1 got 0
I20250115 14:51:53.961618 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_881f3ef58e824b4cb4e1fb3ec4846cc7 [id=0ef6f07cf29e46e69bf2a4449187a4e7]). Expected version 1 got 0
I20250115 14:51:53.961869 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_525bdf5063dd433a8f18376112295a74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:53.962388 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_525bdf5063dd433a8f18376112295a74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:53.962484 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f6205e20a1984ccabccc45b93f9c7fa8
I20250115 14:51:53.964248 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f6205e20a1984ccabccc45b93f9c7fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:53.964509 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet f6205e20a1984ccabccc45b93f9c7fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:53.964571 1167 tablet_replica.cc:331] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:53.964762 1167 raft_consensus.cc:2238] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:53.964808 1165 tablet_replica.cc:331] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:53.965020 1165 raft_consensus.cc:2238] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.965051 1167 pending_rounds.cc:62] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:53.965109 1167 pending_rounds.cc:69] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558314702872576 op_type: NO_OP noop_request { }
I20250115 14:51:53.965221 1165 raft_consensus.cc:2267] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.965251 1167 raft_consensus.cc:2883] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:53.965339 1167 raft_consensus.cc:2267] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:53.965514 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51b5bc2c71cf41809d7057acd47e27be has no consensus state
I20250115 14:51:53.965808 1165 ts_tablet_manager.cc:1905] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:53.965895 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd3e82c5aab54a6ab5449586ad928de5 has no consensus state
I20250115 14:51:53.966434 1167 ts_tablet_manager.cc:1905] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:53.967906 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe0c1c829aff438a94bc41609bdb7841 consensus state has no leader
I20250115 14:51:53.968817 1168 tablet_bootstrap.cc:492] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:53.968986 1168 ts_tablet_manager.cc:1397] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.001s
I20250115 14:51:53.969044 453 master_service.cc:946] ReplaceTablet: received request to replace tablet b027233cfd314d0ebda472b4588d3f00 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:53.969744 1168 raft_consensus.cc:357] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.969965 1168 raft_consensus.cc:383] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.970053 1168 raft_consensus.cc:738] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.970310 1168 consensus_queue.cc:260] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:53.970844 1168 ts_tablet_manager.cc:1428] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:53.971161 1168 tablet_bootstrap.cc:492] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:53.972597 1167 ts_tablet_manager.cc:1918] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:53.972723 1167 log.cc:1198] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f6205e20a1984ccabccc45b93f9c7fa8
I20250115 14:51:53.975160 1162 tablet_bootstrap.cc:492] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:53.975347 1162 ts_tablet_manager.cc:1397] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250115 14:51:53.975683 1168 tablet_bootstrap.cc:654] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.976377 1162 raft_consensus.cc:357] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.976588 1162 raft_consensus.cc:383] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:53.976666 1162 raft_consensus.cc:738] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:53.976917 1162 consensus_queue.cc:260] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.977308 1165 ts_tablet_manager.cc:1918] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:53.978008 1165 log.cc:1198] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f6205e20a1984ccabccc45b93f9c7fa8
I20250115 14:51:53.978678 1167 ts_tablet_manager.cc:1939] T f6205e20a1984ccabccc45b93f9c7fa8 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:53.979617 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f6205e20a1984ccabccc45b93f9c7fa8 (table default.Table_c68a808d4c4744fe92a97753d517180a [id=7222964ef7514b2ba94aa1527810947a]) successfully deleted
I20250115 14:51:53.981420 1164 tablet_bootstrap.cc:654] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.978804 1162 ts_tablet_manager.cc:1428] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:53.983434 1165 ts_tablet_manager.cc:1939] T f6205e20a1984ccabccc45b93f9c7fa8 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:53.984201 1162 tablet_bootstrap.cc:492] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:53.986032 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f6205e20a1984ccabccc45b93f9c7fa8 (table default.Table_c68a808d4c4744fe92a97753d517180a [id=7222964ef7514b2ba94aa1527810947a]) successfully deleted
I20250115 14:51:53.986760 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 715274a681f447c594e8fcd38d6fd609 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.987071 1162 tablet_bootstrap.cc:654] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:53.987071 1166 tablet_replica.cc:331] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:53.987236 1166 raft_consensus.cc:2238] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:53.987419 1166 raft_consensus.cc:2267] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:53.988227 747 tablet_service.cc:1514] Processing DeleteTablet for tablet f6205e20a1984ccabccc45b93f9c7fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:53.989387 1166 ts_tablet_manager.cc:1905] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:53.993846 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 259ac764ca844df9b6c0ce1688a188da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 259ac764ca844df9b6c0ce1688a188da already in progress: creating tablet
I20250115 14:51:53.997778 1357 raft_consensus.cc:491] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:53.997984 1357 raft_consensus.cc:513] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:53.998662 1357 leader_election.cc:290] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:53.999059 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb792ca1b3f54def92f504ec8682d2c7" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:53.999687 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb792ca1b3f54def92f504ec8682d2c7" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:53.999994 848 leader_election.cc:343] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:54.000356 847 leader_election.cc:343] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.000470 847 leader_election.cc:304] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.000955 1357 raft_consensus.cc:2743] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:54.001561 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 24751e606ad94675b650d67a3738481e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.002586 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet f6205e20a1984ccabccc45b93f9c7fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.003305 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet ad67354f35ce4564b88dbdffcabf3564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.004477 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet d77ff794a83348c29391786666987db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.004959 1166 ts_tablet_manager.cc:1918] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.005069 1165 tablet_replica.cc:331] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:54.005277 1165 raft_consensus.cc:2238] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.005481 1165 raft_consensus.cc:2267] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.006263 1165 ts_tablet_manager.cc:1905] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.014096 1165 ts_tablet_manager.cc:1918] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.014232 1165 log.cc:1198] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/24751e606ad94675b650d67a3738481e
I20250115 14:51:54.007869 747 tablet_service.cc:1514] Processing DeleteTablet for tablet c322301320ba425a94d7321c08f7f9e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.014567 1165 ts_tablet_manager.cc:1939] T 24751e606ad94675b650d67a3738481e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:54.014827 1166 log.cc:1198] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/715274a681f447c594e8fcd38d6fd609
I20250115 14:51:54.015395 1166 ts_tablet_manager.cc:1939] T 715274a681f447c594e8fcd38d6fd609 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:54.017935 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet fe0c1c829aff438a94bc41609bdb7841
I20250115 14:51:54.018393 449 catalog_manager.cc:6596] ReplaceTablet: tablet fe0c1c829aff438a94bc41609bdb7841 deleted and replaced by tablet 23cba5cc51d7414a9352caedcc362c9c
I20250115 14:51:54.019235 886 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0c1c829aff438a94bc41609bdb7841 with delete_type TABLET_DATA_DELETED (replaced by tablet 23cba5cc51d7414a9352caedcc362c9c) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.019407 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0c1c829aff438a94bc41609bdb7841 with delete_type TABLET_DATA_DELETED (replaced by tablet 23cba5cc51d7414a9352caedcc362c9c) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.019781 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b027233cfd314d0ebda472b4588d3f00
I20250115 14:51:54.020195 453 catalog_manager.cc:6596] ReplaceTablet: tablet b027233cfd314d0ebda472b4588d3f00 deleted and replaced by tablet 9145bbaf3e234f079e355454f3ce28cb
I20250115 14:51:54.021338 1166 tablet_replica.cc:331] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:54.021522 1166 raft_consensus.cc:2238] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.021668 1166 raft_consensus.cc:2267] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.021471 454 catalog_manager.cc:5526] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:54.021937 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d6a9fb932c5b4545a4609d568f0e5494 (table default.Table_8f6e7d6b890e4a449f170811c030b6f5 [id=3f3fecceceb3464bb12462bb3041b458]). Expected version 2 got 0
I20250115 14:51:54.022195 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bb792ca1b3f54def92f504ec8682d2c7 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]). Expected version 2 got 0
I20250115 14:51:54.022496 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet fad6f9f7f853418e85353edfd38cc80e (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]). Expected version 2 got 0
I20250115 14:51:54.022724 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 2711a7d437f44b2fa11504d8e400e360 (table default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8 [id=a02ee91f154e41fd9ee54b476fd77331]). Expected version 1 got 0
I20250115 14:51:54.022945 454 catalog_manager.cc:5526] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:54.023341 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_881f3ef58e824b4cb4e1fb3ec4846cc7 [id=0ef6f07cf29e46e69bf2a4449187a4e7]). Expected version 1 got 0
I20250115 14:51:54.023561 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 31d95c9359e54f1a88a07d5c5cce7fa1 (table default.Table_9984f5b933e4431ba6201377446f3026 [id=db3a3335dd864c7384b2a414c31a6d2b]). Expected version 1 got 0
I20250115 14:51:54.023595 1166 ts_tablet_manager.cc:1905] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:54.025173 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet fe0c1c829aff438a94bc41609bdb7841 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe0c1c829aff438a94bc41609bdb7841 already in progress: creating tablet
I20250115 14:51:54.025803 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b027233cfd314d0ebda472b4588d3f00 with delete_type TABLET_DATA_DELETED (replaced by tablet 9145bbaf3e234f079e355454f3ce28cb) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.026077 1167 tablet_replica.cc:331] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:54.026201 1165 tablet_replica.cc:331] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:54.026265 1167 raft_consensus.cc:2238] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.026393 1165 raft_consensus.cc:2238] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.026440 1167 raft_consensus.cc:2267] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.026759 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet b027233cfd314d0ebda472b4588d3f00 with delete_type TABLET_DATA_DELETED (replaced by tablet 9145bbaf3e234f079e355454f3ce28cb) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.027045 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 24751e606ad94675b650d67a3738481e (table default.Table_49eb379e719f4677b13269d15ef7efad [id=af78a0b6494d4fd68224992dec097051]) successfully deleted
I20250115 14:51:54.027132 1167 ts_tablet_manager.cc:1905] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:54.027273 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f6205e20a1984ccabccc45b93f9c7fa8 because the tablet was not found. No further retry: Not found: Tablet not found: f6205e20a1984ccabccc45b93f9c7fa8
I20250115 14:51:54.026541 1165 raft_consensus.cc:2267] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.029233 1164 tablet_bootstrap.cc:492] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:54.029455 1164 ts_tablet_manager.cc:1397] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.079s user 0.000s sys 0.003s
I20250115 14:51:54.030061 1165 ts_tablet_manager.cc:1905] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.030436 1164 raft_consensus.cc:357] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.030711 1164 raft_consensus.cc:383] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.030987 1164 raft_consensus.cc:738] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.031371 1164 consensus_queue.cc:260] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.034173 1167 ts_tablet_manager.cc:1918] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.034348 1167 log.cc:1198] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b027233cfd314d0ebda472b4588d3f00
I20250115 14:51:54.034773 1167 ts_tablet_manager.cc:1939] T b027233cfd314d0ebda472b4588d3f00 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:54.036784 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b027233cfd314d0ebda472b4588d3f00 (table default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]) successfully deleted
I20250115 14:51:54.038743 1166 ts_tablet_manager.cc:1918] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.039716 1166 log.cc:1198] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/c322301320ba425a94d7321c08f7f9e6
I20250115 14:51:54.040127 1165 ts_tablet_manager.cc:1918] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.040542 1166 ts_tablet_manager.cc:1939] T c322301320ba425a94d7321c08f7f9e6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:54.044062 1164 ts_tablet_manager.cc:1428] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20250115 14:51:54.047396 1164 tablet_bootstrap.cc:492] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:54.049232 1164 tablet_bootstrap.cc:654] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.040271 1165 log.cc:1198] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ad67354f35ce4564b88dbdffcabf3564
I20250115 14:51:54.051546 1168 tablet_bootstrap.cc:492] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:54.051707 1168 ts_tablet_manager.cc:1397] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.021s
I20250115 14:51:54.052524 747 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0c1c829aff438a94bc41609bdb7841 with delete_type TABLET_DATA_DELETED (replaced by tablet 23cba5cc51d7414a9352caedcc362c9c) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.052577 1168 raft_consensus.cc:357] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.052771 1168 raft_consensus.cc:383] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.052863 1168 raft_consensus.cc:738] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.053099 1168 consensus_queue.cc:260] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.054428 1162 tablet_bootstrap.cc:492] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.054978 747 tablet_service.cc:1514] Processing DeleteTablet for tablet b027233cfd314d0ebda472b4588d3f00 with delete_type TABLET_DATA_DELETED (replaced by tablet 9145bbaf3e234f079e355454f3ce28cb) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.054687 1168 ts_tablet_manager.cc:1428] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:54.055366 1168 tablet_bootstrap.cc:492] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:54.056458 1162 ts_tablet_manager.cc:1397] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.072s user 0.010s sys 0.015s
I20250115 14:51:54.057062 1168 tablet_bootstrap.cc:654] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.057181 912 raft_consensus.cc:1270] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:54.057483 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_600b928b6de9443db6b3c73398af80b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:54.057659 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dc004b8b6d6944e6a55197bf7065f0f7" } new_table_name: "default.Table_4cfdadf8899b4ab9b5400953ec24e992" modify_external_catalogs: true
W20250115 14:51:54.058089 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_600b928b6de9443db6b3c73398af80b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.058458 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_435f7dd182634613bd6116abea02ff0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.058914 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_435f7dd182634613bd6116abea02ff0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.059494 1165 ts_tablet_manager.cc:1939] T ad67354f35ce4564b88dbdffcabf3564 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:54.060557 1165 tablet_replica.cc:331] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:54.060566 776 raft_consensus.cc:3054] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.060741 1165 raft_consensus.cc:2238] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.060904 1165 raft_consensus.cc:2267] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.060844 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f1d3fac40c0e41be85f83948d69ed4bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.061291 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1d3fac40c0e41be85f83948d69ed4bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.061645 1165 ts_tablet_manager.cc:1905] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.063154 1162 raft_consensus.cc:357] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.063354 776 raft_consensus.cc:1270] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:54.063380 1162 raft_consensus.cc:383] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.063732 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f60611e6b8324d44b54218a523f2fedd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.064277 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f60611e6b8324d44b54218a523f2fedd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.065227 1357 raft_consensus.cc:491] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.065398 1357 raft_consensus.cc:513] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.066092 1357 leader_election.cc:290] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.066221 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 715274a681f447c594e8fcd38d6fd609 (table default.Table_87a3775a6a0d41eabb46cb6f64e46c0d [id=d1df7ebdc5994a36800250155aefd024]) successfully deleted
I20250115 14:51:54.066402 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_86054daadda14e4b834b785a68f5e02e" } modify_external_catalogs: true
W20250115 14:51:54.066470 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f6205e20a1984ccabccc45b93f9c7fa8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f6205e20a1984ccabccc45b93f9c7fa8 already in progress: creating tablet
I20250115 14:51:54.066558 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_86054daadda14e4b834b785a68f5e02e" } modify_external_catalogs: true
I20250115 14:51:54.066641 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet c322301320ba425a94d7321c08f7f9e6 (table default.Table_dd152d863f394374a58bf845d3d259e2 [id=d5216eacff2b46ad96642b0ef7fd6403]) successfully deleted
W20250115 14:51:54.066845 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet fe0c1c829aff438a94bc41609bdb7841 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe0c1c829aff438a94bc41609bdb7841 already in progress: creating tablet
W20250115 14:51:54.067050 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b027233cfd314d0ebda472b4588d3f00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b027233cfd314d0ebda472b4588d3f00 already in progress: creating tablet
I20250115 14:51:54.067289 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:40227: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111),
I20250115 14:51:54.067482 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250115 14:51:54.067433 1162 raft_consensus.cc:738] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.067804 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8aff0d541b3147b6acb25a0726018b6f" } modify_external_catalogs: true
I20250115 14:51:54.067989 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8aff0d541b3147b6acb25a0726018b6f" } modify_external_catalogs: true
I20250115 14:51:54.068549 1162 consensus_queue.cc:260] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.068857 1168 tablet_bootstrap.cc:492] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:54.068928 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cca6072e4e4f608c6a45be279304f8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:54.069023 1168 ts_tablet_manager.cc:1397] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20250115 14:51:54.069139 1162 ts_tablet_manager.cc:1428] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250115 14:51:54.069729 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ad67354f35ce4564b88dbdffcabf3564 (table default.Table_bbf01a6c24b9449b8da3ee9428b7bd98 [id=23302b50545244ddb17c074740aacccd]) successfully deleted
I20250115 14:51:54.069717 1168 raft_consensus.cc:357] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.069926 1168 raft_consensus.cc:383] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.070401 1169 consensus_queue.cc:1035] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:54.070549 1168 raft_consensus.cc:738] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.070776 1168 consensus_queue.cc:260] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.070861 1169 consensus_queue.cc:1035] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:54.071336 1162 tablet_bootstrap.cc:492] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:54.072396 847 leader_election.cc:343] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.073154 1162 tablet_bootstrap.cc:654] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.075266 1165 ts_tablet_manager.cc:1918] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.075381 1165 log.cc:1198] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d77ff794a83348c29391786666987db1
I20250115 14:51:54.075713 1165 ts_tablet_manager.cc:1939] T d77ff794a83348c29391786666987db1 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:54.080825 1357 raft_consensus.cc:491] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.080978 1357 raft_consensus.cc:513] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.081624 1357 leader_election.cc:290] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.082266 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3b4117171441a987ac043931fc6897" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:54.082536 1048 raft_consensus.cc:2463] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:54.083714 1162 tablet_bootstrap.cc:492] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.083881 1162 ts_tablet_manager.cc:1397] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20250115 14:51:54.084825 1162 raft_consensus.cc:357] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.085039 1162 raft_consensus.cc:383] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.085116 1162 raft_consensus.cc:738] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.085333 847 leader_election.cc:304] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:54.085331 1162 consensus_queue.cc:260] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.085810 1162 ts_tablet_manager.cc:1428] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:54.086490 1162 tablet_bootstrap.cc:492] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.088354 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3b4117171441a987ac043931fc6897" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:54.089689 1357 raft_consensus.cc:2798] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:54.089798 1357 raft_consensus.cc:491] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:54.089893 1357 raft_consensus.cc:3054] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.090967 1168 ts_tablet_manager.cc:1428] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20250115 14:51:54.091575 1162 tablet_bootstrap.cc:654] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.091581 1168 tablet_bootstrap.cc:492] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:54.093240 1168 tablet_bootstrap.cc:654] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.096318 446 catalog_manager.cc:5526] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:54.096810 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet e0eba208f9be4a588cc81e4d99336eac (table default.Table_2898e18fe4e64a1a92ed3658b91dc502 [id=6d0e848982e245e1870c661f7dc35138]). Expected version 1 got 0
W20250115 14:51:54.098662 848 leader_election.cc:343] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.100733 1162 tablet_bootstrap.cc:492] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.100719 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d77ff794a83348c29391786666987db1 (table default.Table_26cfaab25925425bb14d5384a6ce1827 [id=6c9d98b8d6de494a9c6211022d1fc946]) successfully deleted
I20250115 14:51:54.100888 1162 ts_tablet_manager.cc:1397] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
W20250115 14:51:54.101778 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet fe0c1c829aff438a94bc41609bdb7841 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe0c1c829aff438a94bc41609bdb7841 already in progress: creating tablet
W20250115 14:51:54.103340 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b027233cfd314d0ebda472b4588d3f00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b027233cfd314d0ebda472b4588d3f00 already in progress: creating tablet
I20250115 14:51:54.101622 1162 raft_consensus.cc:357] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.104666 1162 raft_consensus.cc:383] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.104786 1162 raft_consensus.cc:738] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.105271 1168 tablet_bootstrap.cc:492] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:54.105178 1162 consensus_queue.cc:260] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.105854 1162 ts_tablet_manager.cc:1428] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250115 14:51:54.106169 1162 tablet_bootstrap.cc:492] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.106362 1168 ts_tablet_manager.cc:1397] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20250115 14:51:54.107460 1168 raft_consensus.cc:357] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.108125 1162 tablet_bootstrap.cc:654] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.109619 1168 raft_consensus.cc:383] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.109985 1168 raft_consensus.cc:738] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.110448 1168 consensus_queue.cc:260] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.110736 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cca6072e4e4f608c6a45be279304f8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:54.111424 848 leader_election.cc:343] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.111567 848 leader_election.cc:304] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.111805 1209 raft_consensus.cc:2743] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:54.112936 1168 ts_tablet_manager.cc:1428] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:54.113818 1168 tablet_bootstrap.cc:492] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:54.116768 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 7a9e2c053d624b6ba71b031a591475a4 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:54.115458 1357 raft_consensus.cc:513] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.117885 1168 tablet_bootstrap.cc:654] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.118254 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet dd3e82c5aab54a6ab5449586ad928de5
W20250115 14:51:54.119042 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5703fc5995184298bf72f9e43efb41f5 has no consensus state
I20250115 14:51:54.122890 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3b4117171441a987ac043931fc6897" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:54.123169 1048 raft_consensus.cc:3054] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.126109 1048 raft_consensus.cc:2463] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:54.126178 1162 tablet_bootstrap.cc:492] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.126322 1162 ts_tablet_manager.cc:1397] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20250115 14:51:54.126907 847 leader_election.cc:304] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:54.127225 1162 raft_consensus.cc:357] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.127436 1162 raft_consensus.cc:383] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.127516 1162 raft_consensus.cc:738] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.127768 1162 consensus_queue.cc:260] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.128252 1209 raft_consensus.cc:2798] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:54.128322 1162 ts_tablet_manager.cc:1428] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250115 14:51:54.128340 1168 tablet_bootstrap.cc:492] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:54.128418 1209 raft_consensus.cc:695] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:54.128633 1162 tablet_bootstrap.cc:492] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.131608 1162 tablet_bootstrap.cc:654] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:54.133136 984 consensus_peers.cc:487] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:54.133514 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d3b4117171441a987ac043931fc6897" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:54.136147 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fa872457db7140b6a13bfc998e32581b" } modify_external_catalogs: true
I20250115 14:51:54.136394 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fa872457db7140b6a13bfc998e32581b" } modify_external_catalogs: true
I20250115 14:51:54.141045 1164 tablet_bootstrap.cc:492] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:54.141724 1164 ts_tablet_manager.cc:1397] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.094s user 0.000s sys 0.003s
I20250115 14:51:54.143236 1164 raft_consensus.cc:357] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.143437 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ff3a0751d563485db14fc5a44116f2ac" } modify_external_catalogs: true
I20250115 14:51:54.143600 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ff3a0751d563485db14fc5a44116f2ac" } modify_external_catalogs: true
I20250115 14:51:54.144088 1164 raft_consensus.cc:383] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.144192 1164 raft_consensus.cc:738] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.144145 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d89b6c055aff46a79a0e0d1f6fefeb4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.144712 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d89b6c055aff46a79a0e0d1f6fefeb4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.144762 1164 consensus_queue.cc:260] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.145771 1164 ts_tablet_manager.cc:1428] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250115 14:51:54.145816 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:54.146040 1164 tablet_bootstrap.cc:492] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:51:54.146278 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.151857 1357 leader_election.cc:290] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.152570 1164 tablet_bootstrap.cc:654] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.155663 1394 tablet.cc:1721] T 22edaa6e70d942ba98777973476f449d P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:51:54.158901 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1cd00e265b464348bd1df215a1a0d9e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.159377 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cd00e265b464348bd1df215a1a0d9e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.159448 1169 raft_consensus.cc:491] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.159600 1169 raft_consensus.cc:513] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.160224 1169 leader_election.cc:290] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.161644 1164 tablet_bootstrap.cc:492] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:54.161815 1164 ts_tablet_manager.cc:1397] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20250115 14:51:54.162532 1164 raft_consensus.cc:357] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.162750 1164 raft_consensus.cc:383] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.162808 1164 raft_consensus.cc:738] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.163025 1164 consensus_queue.cc:260] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.163478 1164 ts_tablet_manager.cc:1428] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:54.163767 1164 tablet_bootstrap.cc:492] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:54.165352 1164 tablet_bootstrap.cc:654] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.165920 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69aed449a776493b8bdc7a31b47b62e8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:54.168711 984 leader_election.cc:343] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.169198 1162 tablet_bootstrap.cc:492] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.169337 1162 ts_tablet_manager.cc:1397] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.002s
I20250115 14:51:54.170065 1162 raft_consensus.cc:357] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.170250 1162 raft_consensus.cc:383] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.170342 1162 raft_consensus.cc:738] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.170544 1162 consensus_queue.cc:260] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.170671 1169 raft_consensus.cc:491] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.170792 1169 raft_consensus.cc:513] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.171103 1162 ts_tablet_manager.cc:1428] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:54.171391 1162 tablet_bootstrap.cc:492] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.171442 1169 leader_election.cc:290] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.174048 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb792ca1b3f54def92f504ec8682d2c7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:54.174218 1162 tablet_bootstrap.cc:654] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:54.176628 984 leader_election.cc:343] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.180482 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b] alter complete (version 1)
W20250115 14:51:54.188771 984 consensus_peers.cc:487] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:51:54.189711 848 leader_election.cc:343] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.189663 1209 consensus_queue.cc:237] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.190215 913 raft_consensus.cc:1270] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:54.193217 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet bd6d837c64ad4503a224ced67ff1d053
I20250115 14:51:54.193964 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d98a5978bcea4e01af0c694f42d2308a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.194535 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d98a5978bcea4e01af0c694f42d2308a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.196053 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_525bdf5063dd433a8f18376112295a74" } new_table_name: "default.Table_8a7055866139444681c82d83d9af5978" modify_external_catalogs: true
I20250115 14:51:54.196194 1169 consensus_queue.cc:1035] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:54.197432 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9984f5b933e4431ba6201377446f3026" } modify_external_catalogs: true
I20250115 14:51:54.197616 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9984f5b933e4431ba6201377446f3026" } modify_external_catalogs: true
I20250115 14:51:54.199059 453 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 22edaa6e70d942ba98777973476f449d (table default.Table_1b3a658f6eda417cb956eba9cdc5a2af [id=d7ba4cda40964e1d839efe9e0f1615c0]). Expected version 3 got 0
I20250115 14:51:54.200121 1416 mvcc.cc:204] Tried to move back new op lower bound from 7114558316766556160 to 7114558315223764992. Current Snapshot: MvccSnapshot[applied={T|T < 7114558316766556160}]
I20250115 14:51:54.208336 1426 tablet.cc:1721] T 22edaa6e70d942ba98777973476f449d P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:51:54.214607 747 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6d837c64ad4503a224ced67ff1d053 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.214896 749 tablet_service.cc:1467] Processing CreateTablet for tablet f47a8acc3bec46b78725dcc3e675ea74 (DEFAULT_TABLE table=default.Table_aa6a106222694bbfbe6c26b0f8e65396 [id=8725c85d30684db495ec7a20962fd75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.215312 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f47a8acc3bec46b78725dcc3e675ea74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.215801 753 tablet_service.cc:1467] Processing CreateTablet for tablet 51b5bc2c71cf41809d7057acd47e27be (DEFAULT_TABLE table=default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.216122 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51b5bc2c71cf41809d7057acd47e27be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.220523 752 tablet_service.cc:1467] Processing CreateTablet for tablet 0291acf45d75421b8b1c6311560e7f0c (DEFAULT_TABLE table=default.Table_bbe00fc656b44633ae2fd159ae77ffc0 [id=e6f49fcfb7d74c71904b4e6b1bdfee1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.220890 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0291acf45d75421b8b1c6311560e7f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.220940 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_27c15993e89541219ace5c7251953112"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.221460 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27c15993e89541219ace5c7251953112 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.223695 754 tablet_service.cc:1467] Processing CreateTablet for tablet 99a6ee485127484aa71d021edb39ab43 (DEFAULT_TABLE table=default.Table_d3300e90dc3845f78b3c63dfbf681dde [id=cba9cd7a6d434f18b9e621b035a3a6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.224027 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99a6ee485127484aa71d021edb39ab43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.225101 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1230ddf0f50b4c88ae6762341fb2b04a" } new_table_name: "default.Table_5e402e29daa745aa81afd155e5c68bb4" modify_external_catalogs: true
I20250115 14:51:54.228523 741 tablet_service.cc:1467] Processing CreateTablet for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (DEFAULT_TABLE table=default.Table_1230ddf0f50b4c88ae6762341fb2b04a [id=066036f5ecdf48d98dad1e1812fe55cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.228873 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c75a77d7154b5cb05a5d9ac8a20787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.231848 751 tablet_service.cc:1467] Processing CreateTablet for tablet ac57fa232ae4462793b021b60ede0973 (DEFAULT_TABLE table=default.Table_fbd024b67b794f99b66433ffbb89184c [id=b3c701a3bba64b0cbedcf0928b3c873c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.232172 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac57fa232ae4462793b021b60ede0973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.236529 746 tablet_service.cc:1467] Processing CreateTablet for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 (DEFAULT_TABLE table=default.Table_620e744402d54372bb5c38ad4e168dc1 [id=9a5faae475a948528f47d259a547189b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.236845 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73bd0141ccc4ec4b052b18b9fb62bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.237900 1162 tablet_bootstrap.cc:492] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.238078 1162 ts_tablet_manager.cc:1397] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.005s
I20250115 14:51:54.238824 1162 raft_consensus.cc:357] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.239038 1162 raft_consensus.cc:383] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.239101 1162 raft_consensus.cc:738] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.239326 1162 consensus_queue.cc:260] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.239806 1162 ts_tablet_manager.cc:1428] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:54.240113 1162 tablet_bootstrap.cc:492] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.240314 756 tablet_service.cc:1467] Processing CreateTablet for tablet fe696560ba6b4a76ad013d9a3486e398 (DEFAULT_TABLE table=default.Table_eb6649c5d48f43e5959aa73d981a61cc [id=5bccc8cec8fc49d9aed2bc209bdf53cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.240650 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe696560ba6b4a76ad013d9a3486e398. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.241657 1162 tablet_bootstrap.cc:654] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.244575 742 tablet_service.cc:1467] Processing CreateTablet for tablet 5703fc5995184298bf72f9e43efb41f5 (DEFAULT_TABLE table=default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.244997 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5703fc5995184298bf72f9e43efb41f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.248714 743 tablet_service.cc:1467] Processing CreateTablet for tablet a915e0c0a30940ed8a6233f117e2a560 (DEFAULT_TABLE table=default.Table_d5e0fb344ffa468286ab7ab95c01b841 [id=63c4e413417941a9ae951ece417e74ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.249276 744 tablet_service.cc:1467] Processing CreateTablet for tablet 79aad920dd9744d9b87a8234c92232d8 (DEFAULT_TABLE table=default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.249732 1166 tablet_replica.cc:331] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:54.249890 1166 raft_consensus.cc:2238] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.250051 1166 raft_consensus.cc:2267] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.250855 1166 ts_tablet_manager.cc:1905] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.251078 1168 ts_tablet_manager.cc:1397] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20250115 14:51:54.253105 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 51b5bc2c71cf41809d7057acd47e27be (DEFAULT_TABLE table=default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.253436 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51b5bc2c71cf41809d7057acd47e27be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.255625 1162 tablet_bootstrap.cc:492] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.255751 1162 ts_tablet_manager.cc:1397] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250115 14:51:54.256587 1162 raft_consensus.cc:357] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.256781 1162 raft_consensus.cc:383] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.256889 1162 raft_consensus.cc:738] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.256981 1028 tablet_service.cc:1467] Processing CreateTablet for tablet f47a8acc3bec46b78725dcc3e675ea74 (DEFAULT_TABLE table=default.Table_aa6a106222694bbfbe6c26b0f8e65396 [id=8725c85d30684db495ec7a20962fd75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.257165 1162 consensus_queue.cc:260] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.257315 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f47a8acc3bec46b78725dcc3e675ea74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.257728 1162 ts_tablet_manager.cc:1428] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:54.258044 1162 tablet_bootstrap.cc:492] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.260269 1162 tablet_bootstrap.cc:654] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.261082 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (DEFAULT_TABLE table=default.Table_1230ddf0f50b4c88ae6762341fb2b04a [id=066036f5ecdf48d98dad1e1812fe55cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.261397 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c75a77d7154b5cb05a5d9ac8a20787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.264581 1019 tablet_service.cc:1467] Processing CreateTablet for tablet ac57fa232ae4462793b021b60ede0973 (DEFAULT_TABLE table=default.Table_fbd024b67b794f99b66433ffbb89184c [id=b3c701a3bba64b0cbedcf0928b3c873c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.264885 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac57fa232ae4462793b021b60ede0973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.268535 1026 tablet_service.cc:1467] Processing CreateTablet for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 (DEFAULT_TABLE table=default.Table_620e744402d54372bb5c38ad4e168dc1 [id=9a5faae475a948528f47d259a547189b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.256778 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 99a6ee485127484aa71d021edb39ab43 (DEFAULT_TABLE table=default.Table_d3300e90dc3845f78b3c63dfbf681dde [id=cba9cd7a6d434f18b9e621b035a3a6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.271045 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e8aabd67059d46fe880f4b1262b8f9d1" } new_table_name: "default.Table_d0d86551c8474fb98ad74cb89ff84e28" modify_external_catalogs: true
I20250115 14:51:54.275820 749 tablet_service.cc:1467] Processing CreateTablet for tablet f847f92cd122423aa93f1afa701b0bbb (DEFAULT_TABLE table=default.Table_12bf6210790a443dbd238713e2f8a874 [id=1fd54f7dafdc4777bd06982391452889]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.276352 747 tablet_service.cc:1467] Processing CreateTablet for tablet 9f327ffa731c4c949b24a34dc54ae260 (DEFAULT_TABLE table=default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.282418 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a915e0c0a30940ed8a6233f117e2a560. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.283016 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f847f92cd122423aa93f1afa701b0bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.283562 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79aad920dd9744d9b87a8234c92232d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.284046 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f327ffa731c4c949b24a34dc54ae260. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.297137 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 0291acf45d75421b8b1c6311560e7f0c (DEFAULT_TABLE table=default.Table_bbe00fc656b44633ae2fd159ae77ffc0 [id=e6f49fcfb7d74c71904b4e6b1bdfee1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.298558 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a915e0c0a30940ed8a6233f117e2a560 (DEFAULT_TABLE table=default.Table_d5e0fb344ffa468286ab7ab95c01b841 [id=63c4e413417941a9ae951ece417e74ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.298866 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aed449a776493b8bdc7a31b47b62e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.299710 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet c56fe5b5b59b4017be3cb0ac877479b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.299947 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 79aad920dd9744d9b87a8234c92232d8 (DEFAULT_TABLE table=default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.300218 1018 tablet_service.cc:1467] Processing CreateTablet for tablet fe696560ba6b4a76ad013d9a3486e398 (DEFAULT_TABLE table=default.Table_eb6649c5d48f43e5959aa73d981a61cc [id=5bccc8cec8fc49d9aed2bc209bdf53cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.300467 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6d837c64ad4503a224ced67ff1d053 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.300964 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73bd0141ccc4ec4b052b18b9fb62bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.303023 1166 ts_tablet_manager.cc:1918] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.303323 1166 log.cc:1198] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/bd6d837c64ad4503a224ced67ff1d053
I20250115 14:51:54.305030 1165 tablet_replica.cc:331] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:54.306289 1166 ts_tablet_manager.cc:1939] T bd6d837c64ad4503a224ced67ff1d053 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:54.307577 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]) successfully deleted
I20250115 14:51:54.307683 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d3b4117171441a987ac043931fc6897 with delete_type TABLET_DATA_DELETED (replaced by tablet 906fd4d0aed1497ebe0c7316aa6424bb) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.307852 1169 raft_consensus.cc:491] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.300791 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 9f327ffa731c4c949b24a34dc54ae260 (DEFAULT_TABLE table=default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.300678 1014 tablet_service.cc:1467] Processing CreateTablet for tablet f847f92cd122423aa93f1afa701b0bbb (DEFAULT_TABLE table=default.Table_12bf6210790a443dbd238713e2f8a874 [id=1fd54f7dafdc4777bd06982391452889]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:54.310498 983 leader_election.cc:343] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 69aed449a776493b8bdc7a31b47b62e8
I20250115 14:51:54.311187 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 5703fc5995184298bf72f9e43efb41f5 (DEFAULT_TABLE table=default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.311884 1165 raft_consensus.cc:2238] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.312081 1165 raft_consensus.cc:2267] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.312862 1165 ts_tablet_manager.cc:1905] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.313616 1162 tablet_bootstrap.cc:492] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.311028 1169 raft_consensus.cc:513] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.313637 983 leader_election.cc:304] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.315143 1162 ts_tablet_manager.cc:1397] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.022s
W20250115 14:51:54.315263 983 raft_consensus.cc:2681] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:51:54.316227 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99a6ee485127484aa71d021edb39ab43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.317068 1169 leader_election.cc:290] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.321288 1162 raft_consensus.cc:357] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.322117 1162 raft_consensus.cc:383] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.322789 1162 raft_consensus.cc:738] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.323046 1162 consensus_queue.cc:260] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.323567 1162 ts_tablet_manager.cc:1428] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:51:54.323861 1162 tablet_bootstrap.cc:492] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.324795 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5703fc5995184298bf72f9e43efb41f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.325083 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe696560ba6b4a76ad013d9a3486e398. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.326140 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f847f92cd122423aa93f1afa701b0bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.327468 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f327ffa731c4c949b24a34dc54ae260. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.327984 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79aad920dd9744d9b87a8234c92232d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.329967 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a915e0c0a30940ed8a6233f117e2a560. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.330184 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0291acf45d75421b8b1c6311560e7f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.336109 1169 raft_consensus.cc:491] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.348476 1231 raft_consensus.cc:491] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.348651 1231 raft_consensus.cc:513] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.349421 1231 leader_election.cc:290] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.372085 1169 raft_consensus.cc:513] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.373004 1162 tablet_bootstrap.cc:654] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.379384 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d95c9359e54f1a88a07d5c5cce7fa1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:54.382166 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cca6072e4e4f608c6a45be279304f8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:54.385455 984 leader_election.cc:343] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:54.385746 984 leader_election.cc:343] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.387472 1164 tablet_bootstrap.cc:492] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:54.387626 1164 ts_tablet_manager.cc:1397] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.224s user 0.040s sys 0.093s
I20250115 14:51:54.389607 1165 ts_tablet_manager.cc:1918] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.400184 1164 raft_consensus.cc:357] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.400461 1164 raft_consensus.cc:383] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.400894 1164 raft_consensus.cc:738] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.401197 1164 consensus_queue.cc:260] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.402140 1164 ts_tablet_manager.cc:1428] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250115 14:51:54.402419 1164 tablet_bootstrap.cc:492] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:54.406733 1169 leader_election.cc:290] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.411568 1164 tablet_bootstrap.cc:654] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.413398 1164 tablet_bootstrap.cc:492] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:54.413563 1164 ts_tablet_manager.cc:1397] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250115 14:51:54.413847 1162 tablet_bootstrap.cc:492] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.413982 1162 ts_tablet_manager.cc:1397] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.002s
I20250115 14:51:54.414328 1165 log.cc:1198] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/69aed449a776493b8bdc7a31b47b62e8
I20250115 14:51:54.414408 1164 raft_consensus.cc:357] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.414651 1164 raft_consensus.cc:383] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.414717 1165 ts_tablet_manager.cc:1939] T 69aed449a776493b8bdc7a31b47b62e8 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:54.414732 1164 raft_consensus.cc:738] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.414959 1164 consensus_queue.cc:260] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.415467 1164 ts_tablet_manager.cc:1428] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:54.415647 1165 tablet_replica.cc:331] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:54.415724 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 69aed449a776493b8bdc7a31b47b62e8 (table default.Table_68b340d318b14c078d634b22c471d979 [id=e2d228ef5e814fd9bf50e2ed7e89be4d]) successfully deleted
I20250115 14:51:54.415776 1164 tablet_bootstrap.cc:492] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:54.415835 1165 raft_consensus.cc:2238] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.415993 1165 raft_consensus.cc:2267] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.416124 1162 raft_consensus.cc:357] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.416466 1162 raft_consensus.cc:383] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.416610 1162 raft_consensus.cc:738] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.417484 1164 tablet_bootstrap.cc:654] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.417418 1162 consensus_queue.cc:260] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.418505 1162 ts_tablet_manager.cc:1428] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:51:54.419063 1162 tablet_bootstrap.cc:492] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.416496 1165 ts_tablet_manager.cc:1905] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.422976 1162 tablet_bootstrap.cc:654] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.424801 886 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6d837c64ad4503a224ced67ff1d053 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.424916 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad6f9f7f853418e85353edfd38cc80e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:54.431897 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8c81f47cd6e4a308cd62124db38d42f has no consensus state
I20250115 14:51:54.432504 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7a9e2c053d624b6ba71b031a591475a4
I20250115 14:51:54.432904 451 catalog_manager.cc:6596] ReplaceTablet: tablet 7a9e2c053d624b6ba71b031a591475a4 deleted and replaced by tablet d5ffa997ca6c46559830b1dedf2d1a66
I20250115 14:51:54.433177 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f7c3d078ce8a49cfa5965cb29b102e42" } new_table_name: "default.Table_c5f3c652f95240ff9f080165af17f5e6" modify_external_catalogs: true
I20250115 14:51:54.434418 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9bcfab1e1f0349739eff636fb1b04806
I20250115 14:51:54.435079 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ae4cccbf33234ae28a199f7926716b28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.435559 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae4cccbf33234ae28a199f7926716b28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.436498 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b9ce445d334842c8a6bacad9f693feda
I20250115 14:51:54.437053 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9c06e47841fa45c59c310505e26c9361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.437479 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c06e47841fa45c59c310505e26c9361 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:54.438918 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80c75a77d7154b5cb05a5d9ac8a20787 consensus state has no leader
I20250115 14:51:54.439275 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8f6e7d6b890e4a449f170811c030b6f5" } modify_external_catalogs: true
I20250115 14:51:54.439426 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8f6e7d6b890e4a449f170811c030b6f5" } modify_external_catalogs: true
I20250115 14:51:54.440728 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 31d95c9359e54f1a88a07d5c5cce7fa1
I20250115 14:51:54.441347 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_92f683675cf0474db415bd700c1c4178" } new_table_name: "default.Table_992f266d1ab24ee68d346afbfdec91eb" modify_external_catalogs: true
W20250115 14:51:54.444993 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet fad6f9f7f853418e85353edfd38cc80e (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet fad6f9f7f853418e85353edfd38cc80e consensus state has no leader
I20250115 14:51:54.445992 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c3b50255eb3e4642abaa02355b819e25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.446396 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3b50255eb3e4642abaa02355b819e25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:54.448726 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bd4102b1f83e414d93a6a68315e36d16 has no consensus state
I20250115 14:51:54.449110 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cb0308c238d7479dbcc041fe3d6e7eff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.449576 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb0308c238d7479dbcc041fe3d6e7eff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.451620 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3c51ba0db5274d3cafe5c89ecf27a771"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.452059 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c51ba0db5274d3cafe5c89ecf27a771 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.452389 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb792ca1b3f54def92f504ec8682d2c7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:54.453900 1167 tablet_replica.cc:331] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:54.454007 1357 consensus_queue.cc:1035] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.148s
I20250115 14:51:54.454293 913 raft_consensus.cc:2463] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:54.457971 1168 raft_consensus.cc:357] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.465492 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9e2c053d624b6ba71b031a591475a4 with delete_type TABLET_DATA_DELETED (replaced by tablet d5ffa997ca6c46559830b1dedf2d1a66) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.466298 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcfab1e1f0349739eff636fb1b04806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.466331 743 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ce445d334842c8a6bacad9f693feda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:54.468355 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7a9e2c053d624b6ba71b031a591475a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a9e2c053d624b6ba71b031a591475a4 already in progress: creating tablet
W20250115 14:51:54.468600 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b9ce445d334842c8a6bacad9f693feda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9ce445d334842c8a6bacad9f693feda already in progress: creating tablet
W20250115 14:51:54.468816 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9bcfab1e1f0349739eff636fb1b04806 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bcfab1e1f0349739eff636fb1b04806 already in progress: creating tablet
I20250115 14:51:54.471086 1168 raft_consensus.cc:383] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.471196 1168 raft_consensus.cc:738] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.473920 886 tablet_service.cc:1467] Processing CreateTablet for tablet 51b5bc2c71cf41809d7057acd47e27be (DEFAULT_TABLE table=default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.474325 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51b5bc2c71cf41809d7057acd47e27be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.474426 879 tablet_service.cc:1467] Processing CreateTablet for tablet ac57fa232ae4462793b021b60ede0973 (DEFAULT_TABLE table=default.Table_fbd024b67b794f99b66433ffbb89184c [id=b3c701a3bba64b0cbedcf0928b3c873c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.474782 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac57fa232ae4462793b021b60ede0973. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.476536 891 tablet_service.cc:1467] Processing CreateTablet for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 (DEFAULT_TABLE table=default.Table_620e744402d54372bb5c38ad4e168dc1 [id=9a5faae475a948528f47d259a547189b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.476862 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a73bd0141ccc4ec4b052b18b9fb62bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.480401 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad6f9f7f853418e85353edfd38cc80e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:54.480572 884 tablet_service.cc:1467] Processing CreateTablet for tablet fe696560ba6b4a76ad013d9a3486e398 (DEFAULT_TABLE table=default.Table_eb6649c5d48f43e5959aa73d981a61cc [id=5bccc8cec8fc49d9aed2bc209bdf53cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.480669 909 raft_consensus.cc:2463] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:54.480913 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe696560ba6b4a76ad013d9a3486e398. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.481000 890 tablet_service.cc:1467] Processing CreateTablet for tablet 99a6ee485127484aa71d021edb39ab43 (DEFAULT_TABLE table=default.Table_d3300e90dc3845f78b3c63dfbf681dde [id=cba9cd7a6d434f18b9e621b035a3a6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.480967 1168 consensus_queue.cc:260] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.481325 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99a6ee485127484aa71d021edb39ab43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.481508 1168 ts_tablet_manager.cc:1428] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.230s user 0.001s sys 0.001s
I20250115 14:51:54.481827 1168 tablet_bootstrap.cc:492] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:54.484066 1168 tablet_bootstrap.cc:654] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.484359 888 tablet_service.cc:1467] Processing CreateTablet for tablet f47a8acc3bec46b78725dcc3e675ea74 (DEFAULT_TABLE table=default.Table_aa6a106222694bbfbe6c26b0f8e65396 [id=8725c85d30684db495ec7a20962fd75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.484448 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d95c9359e54f1a88a07d5c5cce7fa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.484704 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f47a8acc3bec46b78725dcc3e675ea74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.480921 911 raft_consensus.cc:1270] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250115 14:51:54.487634 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 31d95c9359e54f1a88a07d5c5cce7fa1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31d95c9359e54f1a88a07d5c5cce7fa1 already in progress: creating tablet
I20250115 14:51:54.490155 880 tablet_service.cc:1467] Processing CreateTablet for tablet 0291acf45d75421b8b1c6311560e7f0c (DEFAULT_TABLE table=default.Table_bbe00fc656b44633ae2fd159ae77ffc0 [id=e6f49fcfb7d74c71904b4e6b1bdfee1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.490464 889 tablet_service.cc:1467] Processing CreateTablet for tablet 5703fc5995184298bf72f9e43efb41f5 (DEFAULT_TABLE table=default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.490796 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5703fc5995184298bf72f9e43efb41f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.496521 882 tablet_service.cc:1467] Processing CreateTablet for tablet a915e0c0a30940ed8a6233f117e2a560 (DEFAULT_TABLE table=default.Table_d5e0fb344ffa468286ab7ab95c01b841 [id=63c4e413417941a9ae951ece417e74ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.497013 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a915e0c0a30940ed8a6233f117e2a560. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.501258 885 tablet_service.cc:1467] Processing CreateTablet for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (DEFAULT_TABLE table=default.Table_1230ddf0f50b4c88ae6762341fb2b04a [id=066036f5ecdf48d98dad1e1812fe55cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.501504 1167 raft_consensus.cc:2238] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
W20250115 14:51:54.505271 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d86075ae5b894d5986ac4bd538f4d42a consensus state has no leader
I20250115 14:51:54.509690 1209 raft_consensus.cc:491] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.509825 1167 raft_consensus.cc:2267] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.510726 1357 raft_consensus.cc:491] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.509604 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d95c9359e54f1a88a07d5c5cce7fa1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:54.512270 908 raft_consensus.cc:2463] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:54.513608 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcfab1e1f0349739eff636fb1b04806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.513880 891 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ce445d334842c8a6bacad9f693feda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.511737 1209 raft_consensus.cc:513] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.512565 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cca6072e4e4f608c6a45be279304f8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:54.519958 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0291acf45d75421b8b1c6311560e7f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.524868 1167 ts_tablet_manager.cc:1905] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.524996 879 tablet_service.cc:1514] Processing DeleteTablet for tablet b027233cfd314d0ebda472b4588d3f00 with delete_type TABLET_DATA_DELETED (replaced by tablet 9145bbaf3e234f079e355454f3ce28cb) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.524888 888 tablet_service.cc:1467] Processing CreateTablet for tablet f847f92cd122423aa93f1afa701b0bbb (DEFAULT_TABLE table=default.Table_12bf6210790a443dbd238713e2f8a874 [id=1fd54f7dafdc4777bd06982391452889]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.525592 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d95c9359e54f1a88a07d5c5cce7fa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.525794 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9e2c053d624b6ba71b031a591475a4 with delete_type TABLET_DATA_DELETED (replaced by tablet d5ffa997ca6c46559830b1dedf2d1a66) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.525827 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d3b4117171441a987ac043931fc6897 with delete_type TABLET_DATA_DELETED (replaced by tablet 906fd4d0aed1497ebe0c7316aa6424bb) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.526423 1209 leader_election.cc:290] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.526997 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cca6072e4e4f608c6a45be279304f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.527375 884 tablet_service.cc:1467] Processing CreateTablet for tablet 9f327ffa731c4c949b24a34dc54ae260 (DEFAULT_TABLE table=default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.529227 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f312f1e3a33a404ca7284c713859f156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.531153 887 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f104c1a71404591072e7a36b276b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b3f07ba1d929468eb8eb1f949e9931b9) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.532385 886 tablet_service.cc:1467] Processing CreateTablet for tablet 79aad920dd9744d9b87a8234c92232d8 (DEFAULT_TABLE table=default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.532740 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd72a0d586d4a2c817f6a39daa07063 with delete_type TABLET_DATA_DELETED (replaced by tablet d7309ecb11d844848638c042c9bbe5f1) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.533252 911 raft_consensus.cc:2463] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:54.533823 1357 raft_consensus.cc:513] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.534515 1357 leader_election.cc:290] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.463265 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9e2c053d624b6ba71b031a591475a4 with delete_type TABLET_DATA_DELETED (replaced by tablet d5ffa997ca6c46559830b1dedf2d1a66) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.463467 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcfab1e1f0349739eff636fb1b04806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.463653 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ce445d334842c8a6bacad9f693feda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.463824 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d95c9359e54f1a88a07d5c5cce7fa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:54.485313 984 leader_election.cc:343] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.501003 983 leader_election.cc:304] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.535225 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fa787e364d4ca6a4b756a8f4beba3b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:54.535342 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 50060.9us
I20250115 14:51:54.535501 1046 raft_consensus.cc:2463] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:54.535733 983 leader_election.cc:304] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.515089 1049 raft_consensus.cc:1270] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:54.536441 1231 raft_consensus.cc:2798] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:54.536554 1231 raft_consensus.cc:491] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:54.536633 1231 raft_consensus.cc:3054] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.537976 878 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b with delete_type TABLET_DATA_DELETED (replaced by tablet 79aad920dd9744d9b87a8234c92232d8) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.538455 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80c75a77d7154b5cb05a5d9ac8a20787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.538692 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f847f92cd122423aa93f1afa701b0bbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.538926 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f327ffa731c4c949b24a34dc54ae260. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.538967 1209 raft_consensus.cc:491] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.539297 847 leader_election.cc:304] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:54.539345 1231 raft_consensus.cc:513] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.539948 1231 leader_election.cc:290] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.540374 1225 raft_consensus.cc:2798] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:54.540469 1225 raft_consensus.cc:491] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:54.540539 1225 raft_consensus.cc:3054] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.541371 1357 raft_consensus.cc:2798] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:54.539099 1209 raft_consensus.cc:513] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.543167 1225 raft_consensus.cc:513] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.522634 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0f104c1a71404591072e7a36b276b0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:54.544024 1225 leader_election.cc:290] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.544193 1225 consensus_queue.cc:1035] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:54.544390 1357 raft_consensus.cc:491] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:54.544703 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0f104c1a71404591072e7a36b276b0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:54.545284 1357 raft_consensus.cc:3054] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.546159 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79aad920dd9744d9b87a8234c92232d8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:54.546802 847 leader_election.cc:343] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.547595 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2711a7d437f44b2fa11504d8e400e360" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:54.547657 776 raft_consensus.cc:3054] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.547770 1047 raft_consensus.cc:2463] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:54.551432 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fa787e364d4ca6a4b756a8f4beba3b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:54.552246 1168 tablet_bootstrap.cc:492] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:54.552347 1174 consensus_queue.cc:1035] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
W20250115 14:51:54.554868 848 consensus_peers.cc:487] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:51:54.555051 848 leader_election.cc:343] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.555152 848 leader_election.cc:304] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.555557 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb792ca1b3f54def92f504ec8682d2c7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:54.555615 848 leader_election.cc:343] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.553471 847 leader_election.cc:304] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:54.556236 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2711a7d437f44b2fa11504d8e400e360" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:54.556627 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad6f9f7f853418e85353edfd38cc80e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:54.557337 1174 raft_consensus.cc:2743] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:54.557539 1174 consensus_queue.cc:1035] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:54.557962 1174 raft_consensus.cc:2798] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:54.558071 1174 raft_consensus.cc:491] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:54.558166 1174 raft_consensus.cc:3054] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:54.560369 984 leader_election.cc:343] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:54.560804 984 leader_election.cc:343] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.562937 1209 leader_election.cc:290] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.563261 1168 ts_tablet_manager.cc:1397] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.002s
I20250115 14:51:54.567993 1231 raft_consensus.cc:491] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.568151 1231 raft_consensus.cc:513] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.569015 1231 leader_election.cc:290] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.569631 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd72a0d586d4a2c817f6a39daa07063" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:54.570278 984 leader_election.cc:343] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:54.576656 848 leader_election.cc:343] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.578297 1357 raft_consensus.cc:513] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.580590 1174 raft_consensus.cc:513] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.583335 1174 leader_election.cc:290] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.589644 1168 raft_consensus.cc:357] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.589980 1168 raft_consensus.cc:383] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.590965 1168 raft_consensus.cc:738] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
W20250115 14:51:54.592324 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 56184.6us
I20250115 14:51:54.592576 1357 leader_election.cc:290] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.592571 983 leader_election.cc:304] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.592908 1225 raft_consensus.cc:2798] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:54.593019 1225 raft_consensus.cc:491] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:54.593104 1225 raft_consensus.cc:3054] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.596069 1225 raft_consensus.cc:513] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.596807 1225 leader_election.cc:290] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.597224 983 leader_election.cc:304] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.597203 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d95c9359e54f1a88a07d5c5cce7fa1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:54.597689 984 leader_election.cc:343] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.598133 1231 raft_consensus.cc:2798] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:54.598276 1231 raft_consensus.cc:491] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:54.598529 1164 tablet_bootstrap.cc:492] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:54.598786 1164 ts_tablet_manager.cc:1397] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.183s user 0.022s sys 0.040s
I20250115 14:51:54.598999 1231 raft_consensus.cc:3054] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.600589 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd72a0d586d4a2c817f6a39daa07063" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:54.600100 1164 raft_consensus.cc:357] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.600822 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb792ca1b3f54def92f504ec8682d2c7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:54.600984 1164 raft_consensus.cc:383] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.601027 911 raft_consensus.cc:3054] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.601135 1164 raft_consensus.cc:738] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.601579 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31d95c9359e54f1a88a07d5c5cce7fa1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:54.601845 907 raft_consensus.cc:3054] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.601562 1164 consensus_queue.cc:260] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.603869 911 raft_consensus.cc:2463] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:54.604239 1164 ts_tablet_manager.cc:1428] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:54.604197 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad6f9f7f853418e85353edfd38cc80e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:54.604957 908 raft_consensus.cc:3054] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.600823 912 raft_consensus.cc:2463] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:54.605823 907 raft_consensus.cc:2463] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:54.605885 1164 tablet_bootstrap.cc:492] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:54.612624 1168 consensus_queue.cc:260] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.612627 908 raft_consensus.cc:2463] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:54.614213 1168 ts_tablet_manager.cc:1428] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.050s user 0.000s sys 0.002s
I20250115 14:51:54.614542 1168 tablet_bootstrap.cc:492] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:54.618386 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2711a7d437f44b2fa11504d8e400e360" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:54.618844 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fa787e364d4ca6a4b756a8f4beba3b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:54.623278 1231 raft_consensus.cc:513] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.626425 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cca6072e4e4f608c6a45be279304f8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:54.626636 1231 leader_election.cc:290] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:54.627014 984 leader_election.cc:343] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.629125 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58cca6072e4e4f608c6a45be279304f8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:54.629349 910 raft_consensus.cc:3054] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.629307 983 leader_election.cc:304] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.629804 1231 raft_consensus.cc:2798] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:54.630151 1231 raft_consensus.cc:695] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:54.630481 1231 consensus_queue.cc:237] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.632234 910 raft_consensus.cc:2463] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:54.632769 1162 tablet_bootstrap.cc:492] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
W20250115 14:51:54.632871 848 leader_election.cc:343] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.632943 1162 ts_tablet_manager.cc:1397] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.214s user 0.003s sys 0.000s
W20250115 14:51:54.633116 848 leader_election.cc:343] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.634553 983 leader_election.cc:304] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.634949 983 leader_election.cc:304] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.637569 1169 raft_consensus.cc:2798] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:54.637660 1169 raft_consensus.cc:491] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:54.637744 1169 raft_consensus.cc:3054] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.639714 1173 raft_consensus.cc:491] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.639880 1173 raft_consensus.cc:513] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.640475 1173 leader_election.cc:290] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.640497 1169 raft_consensus.cc:513] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.641103 1169 leader_election.cc:290] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.641260 1169 raft_consensus.cc:2798] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:54.641388 1169 raft_consensus.cc:695] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:54.641423 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd72a0d586d4a2c817f6a39daa07063" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:54.641619 1169 consensus_queue.cc:237] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:54.641901 984 leader_election.cc:343] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.646027 1168 tablet_bootstrap.cc:654] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.646338 1167 ts_tablet_manager.cc:1918] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:54.647202 1167 log.cc:1198] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/bd6d837c64ad4503a224ced67ff1d053
I20250115 14:51:54.650094 983 leader_election.cc:304] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.650066 1167 ts_tablet_manager.cc:1939] T bd6d837c64ad4503a224ced67ff1d053 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:54.650621 1225 raft_consensus.cc:2798] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:54.652683 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]) successfully deleted
W20250115 14:51:54.654525 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9bcfab1e1f0349739eff636fb1b04806 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bcfab1e1f0349739eff636fb1b04806 already in progress: creating tablet
I20250115 14:51:54.655714 1162 raft_consensus.cc:357] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.655997 1162 raft_consensus.cc:383] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:51:54.656200 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b9ce445d334842c8a6bacad9f693feda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9ce445d334842c8a6bacad9f693feda already in progress: creating tablet
W20250115 14:51:54.657433 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 109233us
I20250115 14:51:54.658457 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2711a7d437f44b2fa11504d8e400e360" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:54.658648 1047 raft_consensus.cc:3054] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.658836 1167 tablet_replica.cc:331] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:54.661455 1047 raft_consensus.cc:2463] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:54.661967 1167 raft_consensus.cc:2238] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250115 14:51:54.662837 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b027233cfd314d0ebda472b4588d3f00 because the tablet was not found. No further retry: Not found: Tablet not found: b027233cfd314d0ebda472b4588d3f00
I20250115 14:51:54.664459 1394 tablet.cc:1721] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:54.666559 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1fa787e364d4ca6a4b756a8f4beba3b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:54.666764 1047 raft_consensus.cc:3054] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.669445 1047 raft_consensus.cc:2463] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:54.670065 1164 tablet_bootstrap.cc:654] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.674351 983 leader_election.cc:304] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.675161 1231 raft_consensus.cc:2798] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:54.675339 1231 raft_consensus.cc:695] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:54.676000 1231 consensus_queue.cc:237] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.678546 1162 raft_consensus.cc:738] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.678920 1162 consensus_queue.cc:260] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.679472 1162 ts_tablet_manager.cc:1428] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20250115 14:51:54.679786 1162 tablet_bootstrap.cc:492] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.682783 1471 raft_consensus.cc:491] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.682953 1471 raft_consensus.cc:513] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.683717 1471 leader_election.cc:290] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.684098 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4821224ac8f1472d8b1e8afc06a118a3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:54.684211 710 leader_election.cc:343] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: be18f1b6d8a24fabbe818811ef646c62
W20250115 14:51:54.684708 848 leader_election.cc:343] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.685201 1464 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:40227 every 8 connection(s)
I20250115 14:51:54.686838 1209 raft_consensus.cc:491] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.687049 1209 raft_consensus.cc:513] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.687496 1167 raft_consensus.cc:2267] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.689060 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1a38186c5042ff851f57979ab80dbc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:54.689497 1209 leader_election.cc:290] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.686825 847 leader_election.cc:304] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.687853 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd72a0d586d4a2c817f6a39daa07063" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
W20250115 14:51:54.689644 848 consensus_peers.cc:487] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:54.689630 1167 ts_tablet_manager.cc:1905] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.686648 1221 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:40227
I20250115 14:51:54.696686 1209 raft_consensus.cc:2798] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:54.696851 910 raft_consensus.cc:3054] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:54.698397 1165 ts_tablet_manager.cc:1918] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.698568 1165 log.cc:1198] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/c56fe5b5b59b4017be3cb0ac877479b4
I20250115 14:51:54.699801 910 raft_consensus.cc:2463] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
W20250115 14:51:54.700955 848 leader_election.cc:343] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.701368 1209 raft_consensus.cc:695] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:54.701664 1209 consensus_queue.cc:237] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.704388 1165 ts_tablet_manager.cc:1939] T c56fe5b5b59b4017be3cb0ac877479b4 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:54.705149 1162 tablet_bootstrap.cc:654] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.705950 1225 raft_consensus.cc:695] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:54.706609 847 leader_election.cc:304] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.707216 1468 raft_consensus.cc:2798] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:54.707329 1468 raft_consensus.cc:695] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:54.707588 1468 consensus_queue.cc:237] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.708971 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet c56fe5b5b59b4017be3cb0ac877479b4 (table default.Table_740738b002f34e2dba618cd146ff8dcd [id=3e278eb470174b78a8cdaab8d9ab2464]) successfully deleted
I20250115 14:51:54.714458 983 leader_election.cc:304] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.715008 1165 tablet_replica.cc:331] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:54.718964 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8c90e9dde49d43239bcb743145f4ef07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.719599 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c90e9dde49d43239bcb743145f4ef07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.721014 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b1b29136da42404eb226ca785d5aa7bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.721536 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1b29136da42404eb226ca785d5aa7bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.723801 1480 raft_consensus.cc:491] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.724545 1480 raft_consensus.cc:513] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.727142 1470 raft_consensus.cc:2798] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:54.727244 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b027233cfd314d0ebda472b4588d3f00" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:54.724279 1165 raft_consensus.cc:2238] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.728513 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d6a9fb932c5b4545a4609d568f0e5494
I20250115 14:51:54.728765 1480 leader_election.cc:290] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:54.727314 1470 raft_consensus.cc:695] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:54.729199 1047 raft_consensus.cc:1270] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:54.729933 1165 raft_consensus.cc:2267] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.730978 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a9fb932c5b4545a4609d568f0e5494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.731164 883 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a9fb932c5b4545a4609d568f0e5494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:54.731452 710 leader_election.cc:343] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: be18f1b6d8a24fabbe818811ef646c62
I20250115 14:51:54.731637 710 leader_election.cc:304] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:54.732246 1173 raft_consensus.cc:2743] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:54.732568 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d6a9fb932c5b4545a4609d568f0e5494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6a9fb932c5b4545a4609d568f0e5494 already in progress: creating tablet
I20250115 14:51:54.733474 1471 consensus_queue.cc:1035] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:51:54.735510 984 leader_election.cc:343] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.735941 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a9fb932c5b4545a4609d568f0e5494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:54.737324 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 629873c00550408798b00489ece4ea5f has no consensus state
I20250115 14:51:54.737788 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e6612d24c40f44a780551d89e880543a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.738343 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6612d24c40f44a780551d89e880543a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.739759 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c3fec71fedac4b4c9864b18ea6af10b9" } modify_external_catalogs: true
I20250115 14:51:54.739914 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c3fec71fedac4b4c9864b18ea6af10b9" } modify_external_catalogs: true
I20250115 14:51:54.741475 1164 tablet_bootstrap.cc:492] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:54.741748 1164 ts_tablet_manager.cc:1397] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.136s user 0.009s sys 0.001s
I20250115 14:51:54.743132 1164 raft_consensus.cc:357] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.743489 1164 raft_consensus.cc:383] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.743386 1165 ts_tablet_manager.cc:1905] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.743608 1164 raft_consensus.cc:738] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.744004 1164 consensus_queue.cc:260] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.744421 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1a38186c5042ff851f57979ab80dbc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:54.744729 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4821224ac8f1472d8b1e8afc06a118a3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:54.742923 1225 consensus_queue.cc:237] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.743158 1470 consensus_queue.cc:237] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.746311 1164 ts_tablet_manager.cc:1428] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250115 14:51:54.749367 1164 tablet_bootstrap.cc:492] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:54.751914 1168 tablet_bootstrap.cc:492] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:54.752094 1168 ts_tablet_manager.cc:1397] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.138s user 0.006s sys 0.006s
W20250115 14:51:54.747924 983 leader_election.cc:343] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: b027233cfd314d0ebda472b4588d3f00
I20250115 14:51:54.755410 983 leader_election.cc:304] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.755735 1480 raft_consensus.cc:2743] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:54.756664 1490 tablet.cc:1721] T 22edaa6e70d942ba98777973476f449d P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:51:54.758098 1394 tablet.cc:1721] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:54.759176 1168 raft_consensus.cc:357] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.759393 1168 raft_consensus.cc:383] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.759467 1168 raft_consensus.cc:738] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.759687 1168 consensus_queue.cc:260] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.760228 1168 ts_tablet_manager.cc:1428] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:51:54.760535 1168 tablet_bootstrap.cc:492] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:54.763083 1471 raft_consensus.cc:491] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.763209 1471 raft_consensus.cc:513] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.763867 1471 leader_election.cc:290] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.764601 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b76cb343fb4456ac07fd3c6e9888de" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:54.765275 848 leader_election.cc:343] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.740051 1167 ts_tablet_manager.cc:1918] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:54.766108 1167 log.cc:1198] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/31d95c9359e54f1a88a07d5c5cce7fa1
I20250115 14:51:54.766903 1167 ts_tablet_manager.cc:1939] T 31d95c9359e54f1a88a07d5c5cce7fa1 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:54.768451 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 31d95c9359e54f1a88a07d5c5cce7fa1 (table default.Table_9984f5b933e4431ba6201377446f3026 [id=db3a3335dd864c7384b2a414c31a6d2b]) successfully deleted
I20250115 14:51:54.769271 1165 ts_tablet_manager.cc:1918] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:54.769407 1165 log.cc:1198] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/bd6d837c64ad4503a224ced67ff1d053
I20250115 14:51:54.769331 1167 tablet_replica.cc:331] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:54.773135 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 5027fba8fed34db59b484918256abc45 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:54.772406 1167 raft_consensus.cc:2238] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
W20250115 14:51:54.774132 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7a9e2c053d624b6ba71b031a591475a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a9e2c053d624b6ba71b031a591475a4 already in progress: creating tablet
I20250115 14:51:54.775780 1167 pending_rounds.cc:62] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:54.776980 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 07a63a0a8bb44fe9be0740fc777e610c from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:54.777473 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 7a9e2c053d624b6ba71b031a591475a4 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:54.776590 1167 pending_rounds.cc:69] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558317323890688 op_type: NO_OP noop_request { }
I20250115 14:51:54.778178 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bba588d115d34226951843abdb9b6ebe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:54.778774 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bba588d115d34226951843abdb9b6ebe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:54.778685 847 leader_election.cc:343] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.778909 847 leader_election.cc:304] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:54.779284 847 leader_election.cc:343] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.779392 847 leader_election.cc:304] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.779588 1471 raft_consensus.cc:2743] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:54.779743 1471 raft_consensus.cc:2743] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:54.782649 1394 tablet.cc:1721] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:54.782557 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b76cb343fb4456ac07fd3c6e9888de" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:54.782104 1167 raft_consensus.cc:2883] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:54.783712 1167 raft_consensus.cc:2267] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:54.787603 847 leader_election.cc:343] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.787756 847 leader_election.cc:304] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:54.788074 1471 raft_consensus.cc:2743] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:54.790252 1167 ts_tablet_manager.cc:1905] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.791257 1221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/info.pb
I20250115 14:51:54.795765 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task__Z8ae/build/asan/bin/kudu as pid 1221
I20250115 14:51:54.798353 1164 tablet_bootstrap.cc:654] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.799556 1490 tablet.cc:1721] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:54.808244 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_42f23bc0b4dd4745923028c81cf9ac37" } new_table_name: "default.Table_c5ab1b94f5894ae084dc0539c56b0487" modify_external_catalogs: true
I20250115 14:51:54.809106 450 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet fad6f9f7f853418e85353edfd38cc80e (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]). Expected version 2 got 0
I20250115 14:51:54.809470 450 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet bb792ca1b3f54def92f504ec8682d2c7 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]). Expected version 2 got 0
I20250115 14:51:54.810787 1173 raft_consensus.cc:491] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.810923 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 22edaa6e70d942ba98777973476f449d (table default.Table_1b3a658f6eda417cb956eba9cdc5a2af [id=d7ba4cda40964e1d839efe9e0f1615c0]). Expected version 3 got 0
I20250115 14:51:54.810977 1173 raft_consensus.cc:513] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:54.811065 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e932d5b820f94130967eccce9e1542fa (DEFAULT_TABLE table=default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.811161 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4821224ac8f1472d8b1e8afc06a118a3 (table default.Table_721928c380b840ad84f9f5030b52632f [id=c5f831177088497dbf1fee130e231e37]). Expected version 1 got 0
I20250115 14:51:54.811465 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e932d5b820f94130967eccce9e1542fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.811517 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_1b3a658f6eda417cb956eba9cdc5a2af [id=d7ba4cda40964e1d839efe9e0f1615c0] alter complete (version 3)
I20250115 14:51:54.811656 1173 leader_election.cc:290] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.812422 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d551e62017774245b9704a679dbd6e6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:54.812572 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 (DEFAULT_TABLE table=default.Table_435f7dd182634613bd6116abea02ff0f [id=c8b8a13b708841e2ad2017a4fa7d01d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.812693 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a9fb932c5b4545a4609d568f0e5494" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:54.812934 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:54.812943 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d551e62017774245b9704a679dbd6e6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:54.815292 1495 tablet.cc:1721] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:54.815407 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d0d86551c8474fb98ad74cb89ff84e28" } modify_external_catalogs: true
I20250115 14:51:54.815568 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d0d86551c8474fb98ad74cb89ff84e28" } modify_external_catalogs: true
I20250115 14:51:54.816576 1028 tablet_service.cc:1467] Processing CreateTablet for tablet e7100e6a51ed4e178e86138f10d5ffc2 (DEFAULT_TABLE table=default.Table_600b928b6de9443db6b3c73398af80b7 [id=fa886a7e8fcd43a096892c6435e13c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.816916 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7100e6a51ed4e178e86138f10d5ffc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.818401 1174 raft_consensus.cc:491] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:54.818552 1174 raft_consensus.cc:513] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.819168 1174 leader_election.cc:290] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:54.821214 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f312f1e3a33a404ca7284c713859f156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.821421 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cca6072e4e4f608c6a45be279304f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.824440 749 tablet_service.cc:1467] Processing CreateTablet for tablet e932d5b820f94130967eccce9e1542fa (DEFAULT_TABLE table=default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.824831 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e932d5b820f94130967eccce9e1542fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.825496 1490 tablet.cc:1721] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:54.827893 1501 tablet.cc:1721] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:54.829793 1168 tablet_bootstrap.cc:654] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.829856 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 7233ebf3bae1462d8eeb3669b39944c0 (DEFAULT_TABLE table=default.Table_c75173371e554bdb8d99dcf3cea7c244 [id=8170a41205004f9ca83aa1c0ab7a2502]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.830199 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7233ebf3bae1462d8eeb3669b39944c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.830886 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1cd00e265b464348bd1df215a1a0d9e9" } modify_external_catalogs: true
I20250115 14:51:54.831106 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1cd00e265b464348bd1df215a1a0d9e9" } modify_external_catalogs: true
I20250115 14:51:54.832546 1026 tablet_service.cc:1467] Processing CreateTablet for tablet bd4102b1f83e414d93a6a68315e36d16 (DEFAULT_TABLE table=default.Table_d0d86551c8474fb98ad74cb89ff84e28 [id=becd439cb45d48c3974b9ae72768542a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.832896 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd4102b1f83e414d93a6a68315e36d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.835018 885 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0c1c829aff438a94bc41609bdb7841 with delete_type TABLET_DATA_DELETED (replaced by tablet 23cba5cc51d7414a9352caedcc362c9c) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.835237 883 tablet_service.cc:1467] Processing CreateTablet for tablet e932d5b820f94130967eccce9e1542fa (DEFAULT_TABLE table=default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.835373 884 tablet_service.cc:1467] Processing CreateTablet for tablet f8c81f47cd6e4a308cd62124db38d42f (DEFAULT_TABLE table=default.Table_8a7055866139444681c82d83d9af5978 [id=b6de0dbbced44c83af5f98a9d1020cfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.836375 891 tablet_service.cc:1467] Processing CreateTablet for tablet 0de6f6f014074c3d8775cb5e4121ccb0 (DEFAULT_TABLE table=default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.836541 1019 tablet_service.cc:1467] Processing CreateTablet for tablet b92231509c754e8dbfc564813a76a123 (DEFAULT_TABLE table=default.Table_02593da1cd7245f38f85424d1a4baad9 [id=1b38e621a58442c88f40bb9296563c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.836870 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b92231509c754e8dbfc564813a76a123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.837273 893 tablet_service.cc:1467] Processing CreateTablet for tablet 75788f8eb36b446e927197f624fbfdf5 (DEFAULT_TABLE table=default.Table_d89b6c055aff46a79a0e0d1f6fefeb4b [id=7f3b54e6dae14efcb524f4f03c0a57b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.837973 880 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b76cb343fb4456ac07fd3c6e9888de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.838492 878 tablet_service.cc:1467] Processing CreateTablet for tablet 7233ebf3bae1462d8eeb3669b39944c0 (DEFAULT_TABLE table=default.Table_c75173371e554bdb8d99dcf3cea7c244 [id=8170a41205004f9ca83aa1c0ab7a2502]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.838971 887 tablet_service.cc:1467] Processing CreateTablet for tablet bd4102b1f83e414d93a6a68315e36d16 (DEFAULT_TABLE table=default.Table_d0d86551c8474fb98ad74cb89ff84e28 [id=becd439cb45d48c3974b9ae72768542a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.839423 892 tablet_service.cc:1467] Processing CreateTablet for tablet b92231509c754e8dbfc564813a76a123 (DEFAULT_TABLE table=default.Table_02593da1cd7245f38f85424d1a4baad9 [id=1b38e621a58442c88f40bb9296563c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.839718 882 tablet_service.cc:1467] Processing CreateTablet for tablet a3a7a487a42949bd9641c38d6c1fcfd3 (DEFAULT_TABLE table=default.Table_42f23bc0b4dd4745923028c81cf9ac37 [id=738170c5571a4d958944ed432428e99d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.840248 885 tablet_service.cc:1467] Processing CreateTablet for tablet 629873c00550408798b00489ece4ea5f (DEFAULT_TABLE table=default.Table_992f266d1ab24ee68d346afbfdec91eb [id=dfc41804229046b29a6d1230b41b9fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.840535 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e932d5b820f94130967eccce9e1542fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.840538 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 0de6f6f014074c3d8775cb5e4121ccb0 (DEFAULT_TABLE table=default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.840788 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c81f47cd6e4a308cd62124db38d42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.840854 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de6f6f014074c3d8775cb5e4121ccb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.840727 886 tablet_service.cc:1467] Processing CreateTablet for tablet e7100e6a51ed4e178e86138f10d5ffc2 (DEFAULT_TABLE table=default.Table_600b928b6de9443db6b3c73398af80b7 [id=fa886a7e8fcd43a096892c6435e13c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.841181 888 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 (DEFAULT_TABLE table=default.Table_435f7dd182634613bd6116abea02ff0f [id=c8b8a13b708841e2ad2017a4fa7d01d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.841732 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a7a487a42949bd9641c38d6c1fcfd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.841945 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75788f8eb36b446e927197f624fbfdf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.844429 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7100e6a51ed4e178e86138f10d5ffc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.844574 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 75788f8eb36b446e927197f624fbfdf5 (DEFAULT_TABLE table=default.Table_d89b6c055aff46a79a0e0d1f6fefeb4b [id=7f3b54e6dae14efcb524f4f03c0a57b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.844877 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75788f8eb36b446e927197f624fbfdf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.846915 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de6f6f014074c3d8775cb5e4121ccb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.848711 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 629873c00550408798b00489ece4ea5f (DEFAULT_TABLE table=default.Table_992f266d1ab24ee68d346afbfdec91eb [id=dfc41804229046b29a6d1230b41b9fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.848882 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b92231509c754e8dbfc564813a76a123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.849018 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 629873c00550408798b00489ece4ea5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.849355 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 629873c00550408798b00489ece4ea5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.851089 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd4102b1f83e414d93a6a68315e36d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.851917 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.852823 1029 tablet_service.cc:1467] Processing CreateTablet for tablet a3a7a487a42949bd9641c38d6c1fcfd3 (DEFAULT_TABLE table=default.Table_42f23bc0b4dd4745923028c81cf9ac37 [id=738170c5571a4d958944ed432428e99d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.853147 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a7a487a42949bd9641c38d6c1fcfd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.855391 880 tablet_service.cc:1467] Processing CreateTablet for tablet d4c984613c7d45d5b7c513cd900c9558 (DEFAULT_TABLE table=default.Table_f1d3fac40c0e41be85f83948d69ed4bb [id=2fb684a787e4420aa9218b369eba5cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.855728 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c984613c7d45d5b7c513cd900c9558. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.857648 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b with delete_type TABLET_DATA_DELETED (replaced by tablet 79aad920dd9744d9b87a8234c92232d8) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.857683 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 276abefcab6a422aa6467f00864c01f4 (DEFAULT_TABLE table=default.Table_c3fec71fedac4b4c9864b18ea6af10b9 [id=382fc98d4c614a9fb79d50a9d630b36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.857926 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet b027233cfd314d0ebda472b4588d3f00 with delete_type TABLET_DATA_DELETED (replaced by tablet 9145bbaf3e234f079e355454f3ce28cb) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.858165 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276abefcab6a422aa6467f00864c01f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.858292 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d4c984613c7d45d5b7c513cd900c9558 (DEFAULT_TABLE table=default.Table_f1d3fac40c0e41be85f83948d69ed4bb [id=2fb684a787e4420aa9218b369eba5cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.858590 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c984613c7d45d5b7c513cd900c9558. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.861167 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6f07ef30cac2470d9b2f428338dd37a3 (DEFAULT_TABLE table=default.Table_17b896568dc34b1aaf2b3e197d03fdc4 [id=b9d32cae6ebd42cea68dc15f454318ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.863173 1016 tablet_service.cc:1467] Processing CreateTablet for tablet d97c2f443ab040f9bc1924f45379165e (DEFAULT_TABLE table=default.Table_121cfa7da22441d581d821791e428785 [id=470df20dd8bd47f4acfa2d40f2e034bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.863859 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d97c2f443ab040f9bc1924f45379165e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.868309 1394 tablet.cc:1721] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:54.868455 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 6f07ef30cac2470d9b2f428338dd37a3 (DEFAULT_TABLE table=default.Table_17b896568dc34b1aaf2b3e197d03fdc4 [id=b9d32cae6ebd42cea68dc15f454318ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.869478 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f07ef30cac2470d9b2f428338dd37a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.869577 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 9145bbaf3e234f079e355454f3ce28cb (DEFAULT_TABLE table=default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.871201 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9145bbaf3e234f079e355454f3ce28cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.884550 743 tablet_service.cc:1467] Processing CreateTablet for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 (DEFAULT_TABLE table=default.Table_435f7dd182634613bd6116abea02ff0f [id=c8b8a13b708841e2ad2017a4fa7d01d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.886997 744 tablet_service.cc:1467] Processing CreateTablet for tablet e7100e6a51ed4e178e86138f10d5ffc2 (DEFAULT_TABLE table=default.Table_600b928b6de9443db6b3c73398af80b7 [id=fa886a7e8fcd43a096892c6435e13c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.887367 747 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6d837c64ad4503a224ced67ff1d053 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.887956 756 tablet_service.cc:1467] Processing CreateTablet for tablet 629873c00550408798b00489ece4ea5f (DEFAULT_TABLE table=default.Table_992f266d1ab24ee68d346afbfdec91eb [id=dfc41804229046b29a6d1230b41b9fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.888542 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7233ebf3bae1462d8eeb3669b39944c0 (DEFAULT_TABLE table=default.Table_c75173371e554bdb8d99dcf3cea7c244 [id=8170a41205004f9ca83aa1c0ab7a2502]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.889017 741 tablet_service.cc:1467] Processing CreateTablet for tablet b92231509c754e8dbfc564813a76a123 (DEFAULT_TABLE table=default.Table_02593da1cd7245f38f85424d1a4baad9 [id=1b38e621a58442c88f40bb9296563c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.889887 748 tablet_service.cc:1467] Processing CreateTablet for tablet d4c984613c7d45d5b7c513cd900c9558 (DEFAULT_TABLE table=default.Table_f1d3fac40c0e41be85f83948d69ed4bb [id=2fb684a787e4420aa9218b369eba5cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.890374 755 tablet_service.cc:1467] Processing CreateTablet for tablet 6f07ef30cac2470d9b2f428338dd37a3 (DEFAULT_TABLE table=default.Table_17b896568dc34b1aaf2b3e197d03fdc4 [id=b9d32cae6ebd42cea68dc15f454318ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.890784 1504 sys_catalog.cc:263] Verifying existing consensus state
I20250115 14:51:54.891542 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0c1c829aff438a94bc41609bdb7841" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:54.892139 752 tablet_service.cc:1467] Processing CreateTablet for tablet a3a7a487a42949bd9641c38d6c1fcfd3 (DEFAULT_TABLE table=default.Table_42f23bc0b4dd4745923028c81cf9ac37 [id=738170c5571a4d958944ed432428e99d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.889642 745 tablet_service.cc:1467] Processing CreateTablet for tablet 75788f8eb36b446e927197f624fbfdf5 (DEFAULT_TABLE table=default.Table_d89b6c055aff46a79a0e0d1f6fefeb4b [id=7f3b54e6dae14efcb524f4f03c0a57b1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:54.899730 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet bd6d837c64ad4503a224ced67ff1d053 because the tablet was not found. No further retry: Not found: Tablet not found: bd6d837c64ad4503a224ced67ff1d053
I20250115 14:51:54.900207 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: Bootstrap starting.
I20250115 14:51:54.908322 451 catalog_manager.cc:5526] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:54.908774 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet fad6f9f7f853418e85353edfd38cc80e (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]). Expected version 2 got 0
I20250115 14:51:54.909296 451 catalog_manager.cc:5526] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:54.909663 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet bb792ca1b3f54def92f504ec8682d2c7 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]). Expected version 2 got 0
I20250115 14:51:54.909925 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_881f3ef58e824b4cb4e1fb3ec4846cc7 [id=0ef6f07cf29e46e69bf2a4449187a4e7]). Expected version 1 got 0
I20250115 14:51:54.910219 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 2711a7d437f44b2fa11504d8e400e360 (table default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8 [id=a02ee91f154e41fd9ee54b476fd77331]). Expected version 1 got 0
I20250115 14:51:54.913345 1023 tablet_service.cc:1467] Processing CreateTablet for tablet f8c81f47cd6e4a308cd62124db38d42f (DEFAULT_TABLE table=default.Table_8a7055866139444681c82d83d9af5978 [id=b6de0dbbced44c83af5f98a9d1020cfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.913829 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c81f47cd6e4a308cd62124db38d42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.917160 1162 tablet_bootstrap.cc:492] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.917315 1162 ts_tablet_manager.cc:1397] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.238s user 0.011s sys 0.000s
I20250115 14:51:54.918234 1162 raft_consensus.cc:357] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.918440 1162 raft_consensus.cc:383] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.918530 1162 raft_consensus.cc:738] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.918812 1162 consensus_queue.cc:260] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.919358 1162 ts_tablet_manager.cc:1428] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:54.919631 1162 tablet_bootstrap.cc:492] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.921201 1162 tablet_bootstrap.cc:654] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.924597 1504 log.cc:826] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: Log is configured to *not* fsync() on all Append() calls
I20250115 14:51:54.926851 1162 tablet_bootstrap.cc:492] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:54.927019 1162 ts_tablet_manager.cc:1397] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20250115 14:51:54.927716 1162 raft_consensus.cc:357] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.927943 1162 raft_consensus.cc:383] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:54.928027 1162 raft_consensus.cc:738] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:54.931999 1162 consensus_queue.cc:260] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:54.933863 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 276abefcab6a422aa6467f00864c01f4
I20250115 14:51:54.934567 1162 ts_tablet_manager.cc:1428] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250115 14:51:54.934859 1162 tablet_bootstrap.cc:492] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:54.936031 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 276abefcab6a422aa6467f00864c01f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:54.936451 1162 tablet_bootstrap.cc:654] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:54.868923 881 tablet_service.cc:1467] Processing CreateTablet for tablet 9145bbaf3e234f079e355454f3ce28cb (DEFAULT_TABLE table=default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.955621 879 tablet_service.cc:1467] Processing CreateTablet for tablet d97c2f443ab040f9bc1924f45379165e (DEFAULT_TABLE table=default.Table_121cfa7da22441d581d821791e428785 [id=470df20dd8bd47f4acfa2d40f2e034bd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:54.955664 848 leader_election.cc:343] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:54.955806 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:54.955510 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 629873c00550408798b00489ece4ea5f (table default.Table_992f266d1ab24ee68d346afbfdec91eb [id=dfc41804229046b29a6d1230b41b9fa7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 629873c00550408798b00489ece4ea5f consensus state has no leader
I20250115 14:51:54.956097 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f07ef30cac2470d9b2f428338dd37a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.957321 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 276abefcab6a422aa6467f00864c01f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:54.959498 742 tablet_service.cc:1467] Processing CreateTablet for tablet bd4102b1f83e414d93a6a68315e36d16 (DEFAULT_TABLE table=default.Table_d0d86551c8474fb98ad74cb89ff84e28 [id=becd439cb45d48c3974b9ae72768542a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.959867 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd4102b1f83e414d93a6a68315e36d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.961225 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7233ebf3bae1462d8eeb3669b39944c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.962318 888 tablet_service.cc:1467] Processing CreateTablet for tablet 276abefcab6a422aa6467f00864c01f4 (DEFAULT_TABLE table=default.Table_c3fec71fedac4b4c9864b18ea6af10b9 [id=382fc98d4c614a9fb79d50a9d630b36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.962643 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276abefcab6a422aa6467f00864c01f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.962715 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:54.963836 1095 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:54.964356 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9145bbaf3e234f079e355454f3ce28cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.964363 753 tablet_service.cc:1514] Processing DeleteTablet for tablet be18f1b6d8a24fabbe818811ef646c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.964635 1166 tablet_replica.cc:331] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:54.964816 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d97c2f443ab040f9bc1924f45379165e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.964843 1166 raft_consensus.cc:2238] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:54.965011 1166 raft_consensus.cc:2267] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:54.965523 1166 ts_tablet_manager.cc:1905] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:54.972010 754 tablet_service.cc:1467] Processing CreateTablet for tablet f8c81f47cd6e4a308cd62124db38d42f (DEFAULT_TABLE table=default.Table_8a7055866139444681c82d83d9af5978 [id=b6de0dbbced44c83af5f98a9d1020cfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.972357 746 tablet_service.cc:1467] Processing CreateTablet for tablet 0de6f6f014074c3d8775cb5e4121ccb0 (DEFAULT_TABLE table=default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.972631 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de6f6f014074c3d8775cb5e4121ccb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.976358 750 tablet_service.cc:1467] Processing CreateTablet for tablet 9145bbaf3e234f079e355454f3ce28cb (DEFAULT_TABLE table=default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.976684 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9145bbaf3e234f079e355454f3ce28cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.980508 740 tablet_service.cc:1467] Processing CreateTablet for tablet 276abefcab6a422aa6467f00864c01f4 (DEFAULT_TABLE table=default.Table_c3fec71fedac4b4c9864b18ea6af10b9 [id=382fc98d4c614a9fb79d50a9d630b36d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.981541 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 276abefcab6a422aa6467f00864c01f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.984782 739 tablet_service.cc:1467] Processing CreateTablet for tablet d97c2f443ab040f9bc1924f45379165e (DEFAULT_TABLE table=default.Table_121cfa7da22441d581d821791e428785 [id=470df20dd8bd47f4acfa2d40f2e034bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:54.985224 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d97c2f443ab040f9bc1924f45379165e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.987643 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a9fb932c5b4545a4609d568f0e5494" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:54.989133 747 tablet_service.cc:1514] Processing DeleteTablet for tablet 276abefcab6a422aa6467f00864c01f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:54.991907 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8c81f47cd6e4a308cd62124db38d42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:54.996063 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=5 ignored=0} inserts{seen=5 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates
I20250115 14:51:54.996862 1504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: Bootstrap complete.
I20250115 14:51:54.998389 710 leader_election.cc:304] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:54.998665 1173 raft_consensus.cc:2743] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:54.999256 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 629873c00550408798b00489ece4ea5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.003713 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b92231509c754e8dbfc564813a76a123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.005254 1225 raft_consensus.cc:491] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.005314 1168 tablet_bootstrap.cc:492] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.005462 1168 ts_tablet_manager.cc:1397] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.245s user 0.039s sys 0.075s
I20250115 14:51:55.005431 1225 raft_consensus.cc:513] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.006130 1225 leader_election.cc:290] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.006310 1168 raft_consensus.cc:357] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.006776 1168 raft_consensus.cc:383] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.006742 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0c1c829aff438a94bc41609bdb7841" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:55.006893 1168 raft_consensus.cc:738] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.007119 1168 consensus_queue.cc:260] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:55.007285 984 leader_election.cc:343] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.007826 1168 ts_tablet_manager.cc:1428] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:55.008124 1168 tablet_bootstrap.cc:492] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.008453 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7100e6a51ed4e178e86138f10d5ffc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.010748 1168 tablet_bootstrap.cc:654] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.012356 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a7a487a42949bd9641c38d6c1fcfd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.018023 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75788f8eb36b446e927197f624fbfdf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.021132 1168 tablet_bootstrap.cc:492] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.021317 1168 ts_tablet_manager.cc:1397] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20250115 14:51:55.022138 1168 raft_consensus.cc:357] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.022377 1168 raft_consensus.cc:383] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.022414 1095 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:55.022456 1168 raft_consensus.cc:738] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.022679 1168 consensus_queue.cc:260] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.025138 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7233ebf3bae1462d8eeb3669b39944c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.026940 1168 ts_tablet_manager.cc:1428] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:51:55.027278 1168 tablet_bootstrap.cc:492] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.028849 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4c984613c7d45d5b7c513cd900c9558. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.029675 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f07ef30cac2470d9b2f428338dd37a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.031491 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0c1c829aff438a94bc41609bdb7841" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:55.031718 909 raft_consensus.cc:2463] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:55.032220 983 leader_election.cc:304] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.032613 1225 raft_consensus.cc:2798] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:55.032689 1225 raft_consensus.cc:491] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:55.032768 1225 raft_consensus.cc:3054] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.033324 1167 ts_tablet_manager.cc:1918] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:55.033465 1167 log.cc:1198] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9d3b4117171441a987ac043931fc6897
I20250115 14:51:55.033859 1167 ts_tablet_manager.cc:1939] T 9d3b4117171441a987ac043931fc6897 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.035018 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9d3b4117171441a987ac043931fc6897 (table default.Table_8d37d2aba1094de3a9bc2eabee94643d [id=207a93a004da4fbc91db802c6939059a]) successfully deleted
I20250115 14:51:55.035239 1225 raft_consensus.cc:513] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.036280 1225 leader_election.cc:290] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.036653 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0c1c829aff438a94bc41609bdb7841" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:55.037223 984 leader_election.cc:343] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.038666 1167 tablet_replica.cc:331] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.038900 1167 raft_consensus.cc:2238] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.039070 1167 raft_consensus.cc:2267] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.039616 1167 ts_tablet_manager.cc:1905] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.045130 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0c1c829aff438a94bc41609bdb7841" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:55.045348 909 raft_consensus.cc:3054] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.045990 1167 ts_tablet_manager.cc:1918] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.046137 1167 log.cc:1198] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/58cca6072e4e4f608c6a45be279304f8
I20250115 14:51:55.046463 1167 ts_tablet_manager.cc:1939] T 58cca6072e4e4f608c6a45be279304f8 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.045707 1504 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [term 1 FOLLOWER]: Replica starting. Triggering 2 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:55.047359 1167 tablet_replica.cc:331] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.047459 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 58cca6072e4e4f608c6a45be279304f8 (table default.Table_ca27ab47d077491489383629eaaf1437 [id=5ec280e869eb4ba6bec54fbc22af1409]) successfully deleted
I20250115 14:51:55.047545 1167 raft_consensus.cc:2238] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.047715 1167 raft_consensus.cc:2267] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.048067 909 raft_consensus.cc:2463] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:55.048215 1167 ts_tablet_manager.cc:1905] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.048736 983 leader_election.cc:304] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.049216 1225 raft_consensus.cc:2798] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:55.049402 1225 raft_consensus.cc:695] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:55.049758 1225 consensus_queue.cc:237] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.066145 1166 ts_tablet_manager.cc:1918] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.069384 1501 tablet.cc:1716] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: Skipping requested alter to schema version 2, tablet already version 2
I20250115 14:51:55.071341 1168 tablet_bootstrap.cc:654] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.084378 1164 tablet_bootstrap.cc:492] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:55.084690 1164 ts_tablet_manager.cc:1397] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.336s user 0.006s sys 0.000s
I20250115 14:51:55.086083 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0c1c829aff438a94bc41609bdb7841" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:55.086817 847 leader_election.cc:304] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.087307 1209 raft_consensus.cc:2743] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:55.088194 1394 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.220s user 0.000s sys 0.000s
I20250115 14:51:55.089812 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 23cba5cc51d7414a9352caedcc362c9c (DEFAULT_TABLE table=default.Table_27cd4d793c8f4e77aa531c06dfa74a2d [id=a6994f260e1a4eada78d8c9e9fcf334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.090237 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23cba5cc51d7414a9352caedcc362c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.091689 1394 tablet.cc:1716] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: Skipping requested alter to schema version 2, tablet already version 2
I20250115 14:51:55.092730 1231 raft_consensus.cc:491] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.092907 878 tablet_service.cc:1467] Processing CreateTablet for tablet 23cba5cc51d7414a9352caedcc362c9c (DEFAULT_TABLE table=default.Table_27cd4d793c8f4e77aa531c06dfa74a2d [id=a6994f260e1a4eada78d8c9e9fcf334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.092944 1231 raft_consensus.cc:513] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.093281 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23cba5cc51d7414a9352caedcc362c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.093588 1231 leader_election.cc:290] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.094813 1015 tablet_service.cc:1467] Processing CreateTablet for tablet e3fbde011216426cb41c51fe359dcb5f (DEFAULT_TABLE table=default.Table_1cd00e265b464348bd1df215a1a0d9e9 [id=3fe917c536db4535b79e95a59f9ea8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.095052 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (DEFAULT_TABLE table=default.Table_f60611e6b8324d44b54218a523f2fedd [id=5d05be5a21a44fe0b75c284bf3c83625]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.095152 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3fbde011216426cb41c51fe359dcb5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.095145 1166 log.cc:1198] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/be18f1b6d8a24fabbe818811ef646c62
I20250115 14:51:55.095396 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.095439 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f312f1e3a33a404ca7284c713859f156" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:55.096014 984 leader_election.cc:343] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.096550 1021 tablet_service.cc:1467] Processing CreateTablet for tablet b2c385edf4414157845b971aaf366831 (DEFAULT_TABLE table=default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.096890 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c385edf4414157845b971aaf366831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.099705 1166 ts_tablet_manager.cc:1939] T be18f1b6d8a24fabbe818811ef646c62 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:55.100543 1022 tablet_service.cc:1467] Processing CreateTablet for tablet bc8337d181d44ac7bb82551a677b6831 (DEFAULT_TABLE table=default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.100872 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8337d181d44ac7bb82551a677b6831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.105278 1167 ts_tablet_manager.cc:1918] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.105394 1167 log.cc:1198] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f312f1e3a33a404ca7284c713859f156
I20250115 14:51:55.105742 1167 ts_tablet_manager.cc:1939] T f312f1e3a33a404ca7284c713859f156 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.106586 1167 tablet_replica.cc:331] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.106801 1167 raft_consensus.cc:2238] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.106974 1167 raft_consensus.cc:2267] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.107177 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet be18f1b6d8a24fabbe818811ef646c62 (table default.Table_40664b92268d4ba39f3f5852835f97d2 [id=da39f1615db74469824c3bf8eb74d260]) successfully deleted
W20250115 14:51:55.107416 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 276abefcab6a422aa6467f00864c01f4 because the tablet was not found. No further retry: Not found: Tablet not found: 276abefcab6a422aa6467f00864c01f4
I20250115 14:51:55.107410 890 tablet_service.cc:1467] Processing CreateTablet for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (DEFAULT_TABLE table=default.Table_f60611e6b8324d44b54218a523f2fedd [id=5d05be5a21a44fe0b75c284bf3c83625]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.107600 879 tablet_service.cc:1467] Processing CreateTablet for tablet b2c385edf4414157845b971aaf366831 (DEFAULT_TABLE table=default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.107780 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.108080 888 tablet_service.cc:1467] Processing CreateTablet for tablet bc8337d181d44ac7bb82551a677b6831 (DEFAULT_TABLE table=default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.108439 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8337d181d44ac7bb82551a677b6831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.108534 1165 ts_tablet_manager.cc:1939] T bd6d837c64ad4503a224ced67ff1d053 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:55.109047 1165 tablet_replica.cc:331] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:55.109217 1165 raft_consensus.cc:2238] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.109443 1165 raft_consensus.cc:2267] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.109905 746 tablet_service.cc:1467] Processing CreateTablet for tablet bc8337d181d44ac7bb82551a677b6831 (DEFAULT_TABLE table=default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.110074 1165 ts_tablet_manager.cc:1905] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.110291 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc8337d181d44ac7bb82551a677b6831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.112497 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c385edf4414157845b971aaf366831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.112578 878 tablet_service.cc:1467] Processing CreateTablet for tablet e3fbde011216426cb41c51fe359dcb5f (DEFAULT_TABLE table=default.Table_1cd00e265b464348bd1df215a1a0d9e9 [id=3fe917c536db4535b79e95a59f9ea8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.112907 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3fbde011216426cb41c51fe359dcb5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.112928 749 tablet_service.cc:1467] Processing CreateTablet for tablet e3fbde011216426cb41c51fe359dcb5f (DEFAULT_TABLE table=default.Table_1cd00e265b464348bd1df215a1a0d9e9 [id=3fe917c536db4535b79e95a59f9ea8f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.113267 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3fbde011216426cb41c51fe359dcb5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.115767 754 tablet_service.cc:1467] Processing CreateTablet for tablet b2c385edf4414157845b971aaf366831 (DEFAULT_TABLE table=default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.116125 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c385edf4414157845b971aaf366831. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.116647 740 tablet_service.cc:1467] Processing CreateTablet for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (DEFAULT_TABLE table=default.Table_f60611e6b8324d44b54218a523f2fedd [id=5d05be5a21a44fe0b75c284bf3c83625]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.117035 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.117089 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet bd6d837c64ad4503a224ced67ff1d053 (table default.Table_8aff0d541b3147b6acb25a0726018b6f [id=8767564aae4d4fd198b9c9ea66abe032]) successfully deleted
I20250115 14:51:55.117383 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f312f1e3a33a404ca7284c713859f156 (table default.Table_2752e5f3a4ca4c41a614738b9354d962 [id=1c779c8a53f54707abe1d619b30a912e]) successfully deleted
I20250115 14:51:55.117597 1231 raft_consensus.cc:491] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.117750 1231 raft_consensus.cc:513] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.118435 1231 leader_election.cc:290] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.119201 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0f104c1a71404591072e7a36b276b0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:55.119756 984 leader_election.cc:343] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.125849 1167 ts_tablet_manager.cc:1905] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:55.126803 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3a7a487a42949bd9641c38d6c1fcfd3 consensus state has no leader
I20250115 14:51:55.127256 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5027fba8fed34db59b484918256abc45
I20250115 14:51:55.127799 453 catalog_manager.cc:6596] ReplaceTablet: tablet 5027fba8fed34db59b484918256abc45 deleted and replaced by tablet 37ad2b7350294375bd30fb4b46906491
I20250115 14:51:55.129067 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9502b06eea13418c8ce173b8f4b5ec08" } modify_external_catalogs: true
I20250115 14:51:55.129254 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9502b06eea13418c8ce173b8f4b5ec08" } modify_external_catalogs: true
I20250115 14:51:55.130446 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 5027fba8fed34db59b484918256abc45 with delete_type TABLET_DATA_DELETED (replaced by tablet 37ad2b7350294375bd30fb4b46906491) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.131140 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d89b6c055aff46a79a0e0d1f6fefeb4b" } new_table_name: "default.Table_0ed87b7187f342c8904d2f6d559034e0" modify_external_catalogs: true
I20250115 14:51:55.131531 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 5027fba8fed34db59b484918256abc45 with delete_type TABLET_DATA_DELETED (replaced by tablet 37ad2b7350294375bd30fb4b46906491) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.128170 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2e17c919cc6f4992b7d7b764ba964949" } new_table_name: "default.Table_1213cde4dcb6474487bba376e4516b9c" modify_external_catalogs: true
I20250115 14:51:55.133708 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_910803aa49a042acba7e5a6a65b2e85d" } new_table_name: "default.Table_3fc104bc2f494a76b0f04ab5f41510bb" modify_external_catalogs: true
I20250115 14:51:55.136775 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_02294519a7244bd6a9f4741db4a92d58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.137305 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02294519a7244bd6a9f4741db4a92d58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.139349 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet bc8337d181d44ac7bb82551a677b6831
I20250115 14:51:55.139906 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 7a9e2c053d624b6ba71b031a591475a4 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.140388 452 master_service.cc:946] ReplaceTablet: received request to replace tablet f55c75e5febb407398b0bc06a1db9b1a from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.141898 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0f104c1a71404591072e7a36b276b0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:51:55.142460 983 leader_election.cc:343] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: Tablet not RUNNING: STOPPED
I20250115 14:51:55.142613 983 leader_election.cc:304] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:55.142988 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b90205ef15f470a99c013e7f2cf99c6 consensus state has no leader
I20250115 14:51:55.143301 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c5f3c652f95240ff9f080165af17f5e6" } modify_external_catalogs: true
I20250115 14:51:55.143473 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c5f3c652f95240ff9f080165af17f5e6" } modify_external_catalogs: true
I20250115 14:51:55.144510 1225 raft_consensus.cc:2743] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:55.145567 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet bd4102b1f83e414d93a6a68315e36d16
W20250115 14:51:55.145641 983 leader_election.cc:343] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: f312f1e3a33a404ca7284c713859f156
I20250115 14:51:55.145656 1167 ts_tablet_manager.cc:1918] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.145789 1167 log.cc:1198] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ea0f104c1a71404591072e7a36b276b0
I20250115 14:51:55.145774 983 leader_election.cc:304] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.146111 1167 ts_tablet_manager.cc:1939] T ea0f104c1a71404591072e7a36b276b0 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.146260 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f88c333bf687464cb16fb61569488e70" } new_table_name: "default.Table_90f2e2c555f84fe39b5b478581ef0e44" modify_external_catalogs: true
I20250115 14:51:55.146708 1225 raft_consensus.cc:2743] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:55.146994 888 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4102b1f83e414d93a6a68315e36d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.147061 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet ea0f104c1a71404591072e7a36b276b0 (table default.Table_30d9bde332c24d2a94267ffd2605a7cf [id=717e74f1625e4a5a92c254f204df1629]) successfully deleted
W20250115 14:51:55.147832 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75788f8eb36b446e927197f624fbfdf5 consensus state has no leader
I20250115 14:51:55.147832 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4102b1f83e414d93a6a68315e36d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.148067 879 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8337d181d44ac7bb82551a677b6831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.148337 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8337d181d44ac7bb82551a677b6831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.148833 959 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:55.148187 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c14780544d7c4f43899713097a27848c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.149439 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c14780544d7c4f43899713097a27848c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.151387 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e3fbde011216426cb41c51fe359dcb5f
I20250115 14:51:55.152323 879 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fbde011216426cb41c51fe359dcb5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.152201 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8f5c258dbdc04de79751c3a1b995e0fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.152608 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fbde011216426cb41c51fe359dcb5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:55.152724 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f5c258dbdc04de79751c3a1b995e0fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:55.156081 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b60ff80679a470cb4bdced71cc14195 consensus state has no leader
I20250115 14:51:55.156585 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7171658b8daf4ee3868f00408ce43d84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.157050 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7171658b8daf4ee3868f00408ce43d84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.159688 1167 tablet_replica.cc:331] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.159987 1167 raft_consensus.cc:2238] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.159791 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.160220 1167 raft_consensus.cc:2267] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.161554 1162 tablet_bootstrap.cc:492] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
W20250115 14:51:55.161648 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.161721 1162 ts_tablet_manager.cc:1397] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.227s user 0.027s sys 0.028s
I20250115 14:51:55.162519 1162 raft_consensus.cc:357] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.162739 1162 raft_consensus.cc:383] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.162823 1162 raft_consensus.cc:738] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.163060 1162 consensus_queue.cc:260] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.163628 1162 ts_tablet_manager.cc:1428] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:55.163717 739 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fbde011216426cb41c51fe359dcb5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.165668 1167 ts_tablet_manager.cc:1905] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.165690 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]). Expected version 1 got 0
I20250115 14:51:55.167294 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 99a6ee485127484aa71d021edb39ab43 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.170401 1162 tablet_bootstrap.cc:492] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.170421 455 catalog_manager.cc:5526] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:55.170827 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 2711a7d437f44b2fa11504d8e400e360 (table default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8 [id=a02ee91f154e41fd9ee54b476fd77331]). Expected version 1 got 0
I20250115 14:51:55.171100 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ab1a38186c5042ff851f57979ab80dbc (table default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]). Expected version 2 got 0
I20250115 14:51:55.172196 1162 tablet_bootstrap.cc:654] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.174867 1173 raft_consensus.cc:491] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.175073 1173 raft_consensus.cc:513] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.175457 1165 ts_tablet_manager.cc:1918] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.175575 1165 log.cc:1198] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9d3b4117171441a987ac043931fc6897
I20250115 14:51:55.175720 1173 leader_election.cc:290] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:55.175951 1165 ts_tablet_manager.cc:1939] T 9d3b4117171441a987ac043931fc6897 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:51:55.176888 710 leader_election.cc:343] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 6326b1526191464485855b657e34a2b9
I20250115 14:51:55.177227 750 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4102b1f83e414d93a6a68315e36d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:55.177482 710 leader_election.cc:343] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6326b1526191464485855b657e34a2b9
I20250115 14:51:55.177767 743 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8337d181d44ac7bb82551a677b6831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.179221 1167 ts_tablet_manager.cc:1918] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:55.180454 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5ffa997ca6c46559830b1dedf2d1a66 has no consensus state
I20250115 14:51:55.181366 446 master_service.cc:946] ReplaceTablet: received request to replace tablet d86075ae5b894d5986ac4bd538f4d42a from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.181661 1167 log.cc:1198] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/3fd72a0d586d4a2c817f6a39daa07063
I20250115 14:51:55.181759 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f55c75e5febb407398b0bc06a1db9b1a
I20250115 14:51:55.182242 1167 ts_tablet_manager.cc:1939] T 3fd72a0d586d4a2c817f6a39daa07063 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.183204 452 catalog_manager.cc:6596] ReplaceTablet: tablet f55c75e5febb407398b0bc06a1db9b1a deleted and replaced by tablet 0e924821d12045f6bc658461675aead4
I20250115 14:51:55.183243 879 tablet_service.cc:1514] Processing DeleteTablet for tablet f55c75e5febb407398b0bc06a1db9b1a with delete_type TABLET_DATA_DELETED (replaced by tablet 0e924821d12045f6bc658461675aead4) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.183596 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_70c7c9d33ea742618f8a32464ed92f8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.184248 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70c7c9d33ea742618f8a32464ed92f8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.184423 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 3fd72a0d586d4a2c817f6a39daa07063 (table default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]) successfully deleted
I20250115 14:51:55.185007 1167 tablet_replica.cc:331] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.185901 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d3b142be26bc4ef59f6c07abad93e201"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.186564 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3b142be26bc4ef59f6c07abad93e201 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.186825 1167 raft_consensus.cc:2238] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:55.187192 1167 pending_rounds.cc:62] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:55.187525 1167 pending_rounds.cc:69] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558319444922368 op_type: NO_OP noop_request { }
I20250115 14:51:55.188524 1167 raft_consensus.cc:2883] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:55.188870 1167 raft_consensus.cc:2267] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.189466 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_23f12c7168a84d5390b6d4ad9ed07ea1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.189901 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23f12c7168a84d5390b6d4ad9ed07ea1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.190331 1167 ts_tablet_manager.cc:1905] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.177939 710 leader_election.cc:304] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:55.191883 1173 raft_consensus.cc:2743] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:55.192162 739 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a9fb932c5b4545a4609d568f0e5494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.178360 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5027fba8fed34db59b484918256abc45 with delete_type TABLET_DATA_DELETED (replaced by tablet 37ad2b7350294375bd30fb4b46906491) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.192385 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f55c75e5febb407398b0bc06a1db9b1a with delete_type TABLET_DATA_DELETED (replaced by tablet 0e924821d12045f6bc658461675aead4) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.193086 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 6326b1526191464485855b657e34a2b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.194536 1504 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc2e23c560f4096be7b14006bdfc525, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.182548 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet f55c75e5febb407398b0bc06a1db9b1a with delete_type TABLET_DATA_DELETED (replaced by tablet 0e924821d12045f6bc658461675aead4) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.197628 1504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:51:55.198642 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9d3b4117171441a987ac043931fc6897 (table default.Table_8d37d2aba1094de3a9bc2eabee94643d [id=207a93a004da4fbc91db802c6939059a]) successfully deleted
W20250115 14:51:55.199678 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7a9e2c053d624b6ba71b031a591475a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a9e2c053d624b6ba71b031a591475a4 already in progress: creating tablet
I20250115 14:51:55.203977 1167 ts_tablet_manager.cc:1918] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250115 14:51:55.204962 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e3fbde011216426cb41c51fe359dcb5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3fbde011216426cb41c51fe359dcb5f already in progress: creating tablet
I20250115 14:51:55.205473 1167 log.cc:1198] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d1fa787e364d4ca6a4b756a8f4beba3b
I20250115 14:51:55.206043 1167 ts_tablet_manager.cc:1939] T d1fa787e364d4ca6a4b756a8f4beba3b P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.208315 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d1fa787e364d4ca6a4b756a8f4beba3b (table default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]) successfully deleted
I20250115 14:51:55.208743 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 99a6ee485127484aa71d021edb39ab43
I20250115 14:51:55.208892 1167 tablet_replica.cc:331] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.209088 1167 raft_consensus.cc:2238] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:55.209259 449 catalog_manager.cc:6596] ReplaceTablet: tablet 99a6ee485127484aa71d021edb39ab43 deleted and replaced by tablet 00adfe53c726428b9ac2fcb47756b3de
I20250115 14:51:55.209412 1167 raft_consensus.cc:2267] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.209678 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ad45cc2411ad4ee1a65b9db7a0a6e4b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.210266 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad45cc2411ad4ee1a65b9db7a0a6e4b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.209982 1521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:55.211951 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a6ee485127484aa71d021edb39ab43 with delete_type TABLET_DATA_DELETED (replaced by tablet 00adfe53c726428b9ac2fcb47756b3de) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.212649 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a6ee485127484aa71d021edb39ab43 with delete_type TABLET_DATA_DELETED (replaced by tablet 00adfe53c726428b9ac2fcb47756b3de) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.212829 1167 ts_tablet_manager.cc:1905] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.213303 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d86075ae5b894d5986ac4bd538f4d42a
I20250115 14:51:55.214039 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666" } modify_external_catalogs: true
I20250115 14:51:55.214226 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666" } modify_external_catalogs: true
I20250115 14:51:55.211954 1521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:51:55.215008 879 tablet_service.cc:1514] Processing DeleteTablet for tablet d86075ae5b894d5986ac4bd538f4d42a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.215080 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d86075ae5b894d5986ac4bd538f4d42a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.216930 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_2898e18fe4e64a1a92ed3658b91dc502 [id=6d0e848982e245e1870c661f7dc35138] alter complete (version 1)
I20250115 14:51:55.219313 1166 tablet_replica.cc:331] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:55.219496 1166 raft_consensus.cc:2238] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.219702 1166 raft_consensus.cc:2267] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.220446 751 tablet_service.cc:1467] Processing CreateTablet for tablet 23cba5cc51d7414a9352caedcc362c9c (DEFAULT_TABLE table=default.Table_27cd4d793c8f4e77aa531c06dfa74a2d [id=a6994f260e1a4eada78d8c9e9fcf334e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.220664 1166 ts_tablet_manager.cc:1905] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.221887 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23cba5cc51d7414a9352caedcc362c9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.225630 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a6ee485127484aa71d021edb39ab43 with delete_type TABLET_DATA_DELETED (replaced by tablet 00adfe53c726428b9ac2fcb47756b3de) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.225999 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d86075ae5b894d5986ac4bd538f4d42a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.227337 1166 ts_tablet_manager.cc:1918] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:55.225478 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet bd4102b1f83e414d93a6a68315e36d16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4102b1f83e414d93a6a68315e36d16 already in progress: creating tablet
I20250115 14:51:55.228799 1166 log.cc:1198] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d6a9fb932c5b4545a4609d568f0e5494
I20250115 14:51:55.229125 1164 raft_consensus.cc:357] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.229995 1164 raft_consensus.cc:383] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.230139 1164 raft_consensus.cc:738] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.230448 1164 consensus_queue.cc:260] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.231066 1164 ts_tablet_manager.cc:1428] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.146s user 0.002s sys 0.000s
I20250115 14:51:55.231436 1164 tablet_bootstrap.cc:492] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:55.233100 1504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 14:51:55.233578 1164 tablet_bootstrap.cc:654] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:55.235836 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9bcfab1e1f0349739eff636fb1b04806 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bcfab1e1f0349739eff636fb1b04806 already in progress: creating tablet
W20250115 14:51:55.236558 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b9ce445d334842c8a6bacad9f693feda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9ce445d334842c8a6bacad9f693feda already in progress: creating tablet
I20250115 14:51:55.236711 1166 ts_tablet_manager.cc:1939] T d6a9fb932c5b4545a4609d568f0e5494 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:55.236900 1165 tablet_replica.cc:331] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:55.237179 1165 raft_consensus.cc:2238] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:55.237504 1165 pending_rounds.cc:62] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:55.237648 1165 pending_rounds.cc:69] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558319174574080 op_type: NO_OP noop_request { }
I20250115 14:51:55.237896 1165 raft_consensus.cc:2883] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:55.238078 1165 raft_consensus.cc:2267] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.238205 1164 tablet_bootstrap.cc:492] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:55.238374 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d6a9fb932c5b4545a4609d568f0e5494 (table default.Table_8f6e7d6b890e4a449f170811c030b6f5 [id=3f3fecceceb3464bb12462bb3041b458]) successfully deleted
I20250115 14:51:55.238412 1164 ts_tablet_manager.cc:1397] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250115 14:51:55.240648 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 13c00f03edc74af8b084bb236487a347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.242403 1167 ts_tablet_manager.cc:1918] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:51:55.243036 1164 raft_consensus.cc:357] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.244537 1164 raft_consensus.cc:383] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.244905 1164 raft_consensus.cc:738] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.245851 1164 consensus_queue.cc:260] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.247152 1164 ts_tablet_manager.cc:1428] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250115 14:51:55.248386 1164 tablet_bootstrap.cc:492] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:55.248654 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:55.250852 1164 tablet_bootstrap.cc:654] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.251699 1167 log.cc:1198] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d6a9fb932c5b4545a4609d568f0e5494
I20250115 14:51:55.252439 1167 ts_tablet_manager.cc:1939] T d6a9fb932c5b4545a4609d568f0e5494 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.254786 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d6a9fb932c5b4545a4609d568f0e5494 (table default.Table_8f6e7d6b890e4a449f170811c030b6f5 [id=3f3fecceceb3464bb12462bb3041b458]) successfully deleted
I20250115 14:51:55.255574 1167 tablet_replica.cc:331] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.256031 1167 raft_consensus.cc:2238] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.256891 1167 raft_consensus.cc:2267] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.257790 1167 ts_tablet_manager.cc:1905] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.259948 1162 tablet_bootstrap.cc:492] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.260118 1162 ts_tablet_manager.cc:1397] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.002s
I20250115 14:51:55.260892 1162 raft_consensus.cc:357] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.261135 1162 raft_consensus.cc:383] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.261216 1162 raft_consensus.cc:738] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.261488 1162 consensus_queue.cc:260] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.262586 1164 tablet_bootstrap.cc:492] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:55.264199 1164 ts_tablet_manager.cc:1397] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20250115 14:51:55.265264 1164 raft_consensus.cc:357] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.265522 1165 ts_tablet_manager.cc:1905] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.266633 1162 ts_tablet_manager.cc:1428] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250115 14:51:55.266950 1162 tablet_bootstrap.cc:492] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.267120 1168 tablet_bootstrap.cc:492] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.267467 1168 ts_tablet_manager.cc:1397] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.240s user 0.020s sys 0.028s
I20250115 14:51:55.268345 1168 raft_consensus.cc:357] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.268610 1168 raft_consensus.cc:383] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.268770 1162 tablet_bootstrap.cc:654] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.268764 1168 raft_consensus.cc:738] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.269127 1168 consensus_queue.cc:260] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.275449 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_56a365aae805419683d1f4dfd97951cc [id=24d4e203bce64fcdbd5b6fa30a99e6b2] alter complete (version 1)
I20250115 14:51:55.280349 1168 ts_tablet_manager.cc:1428] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250115 14:51:55.280897 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0c1c829aff438a94bc41609bdb7841 with delete_type TABLET_DATA_DELETED (replaced by tablet 23cba5cc51d7414a9352caedcc362c9c) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.281136 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd72a0d586d4a2c817f6a39daa07063 with delete_type TABLET_DATA_DELETED (replaced by tablet d7309ecb11d844848638c042c9bbe5f1) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.287035 1164 raft_consensus.cc:383] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.287459 1168 tablet_bootstrap.cc:492] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.287608 1164 raft_consensus.cc:738] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.289669 1022 tablet_service.cc:1467] Processing CreateTablet for tablet caed629878594954842dfc6a40cc8b6d (DEFAULT_TABLE table=default.Table_b1b29136da42404eb226ca785d5aa7bc [id=b6aeb933c5e64e88a6d794a98650acbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.290055 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caed629878594954842dfc6a40cc8b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.290086 1168 tablet_bootstrap.cc:654] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.291074 1164 consensus_queue.cc:260] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.291669 1164 ts_tablet_manager.cc:1428] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20250115 14:51:55.292019 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a38a6bc32314396b497e7ae166a63f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.293212 751 tablet_service.cc:1467] Processing CreateTablet for tablet caed629878594954842dfc6a40cc8b6d (DEFAULT_TABLE table=default.Table_b1b29136da42404eb226ca785d5aa7bc [id=b6aeb933c5e64e88a6d794a98650acbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.297917 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9aeca9975f1945ad956f47db1ea16a73
I20250115 14:51:55.299180 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aeca9975f1945ad956f47db1ea16a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.299275 1167 ts_tablet_manager.cc:1918] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.299290 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aeca9975f1945ad956f47db1ea16a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.299366 1167 log.cc:1198] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/fe0c1c829aff438a94bc41609bdb7841
I20250115 14:51:55.299466 1162 tablet_bootstrap.cc:492] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.299605 1162 ts_tablet_manager.cc:1397] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.000s
I20250115 14:51:55.300060 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caed629878594954842dfc6a40cc8b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.300694 1162 raft_consensus.cc:357] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.300904 1162 raft_consensus.cc:383] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.300999 1162 raft_consensus.cc:738] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.299703 1167 ts_tablet_manager.cc:1939] T fe0c1c829aff438a94bc41609bdb7841 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.301244 1162 consensus_queue.cc:260] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.302124 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet fe0c1c829aff438a94bc41609bdb7841 (table default.Table_27cd4d793c8f4e77aa531c06dfa74a2d [id=a6994f260e1a4eada78d8c9e9fcf334e]) successfully deleted
I20250115 14:51:55.302198 1167 tablet_replica.cc:331] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.302398 1167 raft_consensus.cc:2238] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.302663 1167 raft_consensus.cc:2267] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.303428 1167 ts_tablet_manager.cc:1905] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.304692 1162 ts_tablet_manager.cc:1428] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:51:55.304960 1164 tablet_bootstrap.cc:492] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:55.307377 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]). Expected version 2 got 0
I20250115 14:51:55.307837 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_881f3ef58e824b4cb4e1fb3ec4846cc7 [id=0ef6f07cf29e46e69bf2a4449187a4e7] alter complete (version 1)
I20250115 14:51:55.309772 879 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a9fb932c5b4545a4609d568f0e5494 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.310161 888 tablet_service.cc:1514] Processing DeleteTablet for tablet d243daba84ff4befa0c4e7bac5fe7476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.310158 879 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b with delete_type TABLET_DATA_DELETED (replaced by tablet 79aad920dd9744d9b87a8234c92232d8) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.310818 1164 tablet_bootstrap.cc:654] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.312853 1162 tablet_bootstrap.cc:492] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.313567 1168 tablet_bootstrap.cc:492] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.314599 1162 tablet_bootstrap.cc:654] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.314889 1168 ts_tablet_manager.cc:1397] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20250115 14:51:55.315023 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 2711a7d437f44b2fa11504d8e400e360 (table default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8 [id=a02ee91f154e41fd9ee54b476fd77331]). Expected version 1 got 0
I20250115 14:51:55.315833 1168 raft_consensus.cc:357] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.316861 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ae60842a2d4486eb7651c3f710ee1e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.317512 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b76cb343fb4456ac07fd3c6e9888de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.318332 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ae60842a2d4486eb7651c3f710ee1e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.318586 1168 raft_consensus.cc:383] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.318698 1168 raft_consensus.cc:738] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.319020 1168 consensus_queue.cc:260] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.319950 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f104c1a71404591072e7a36b276b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b3f07ba1d929468eb8eb1f949e9931b9) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.322194 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 745153c106844076bb020f9ed026c97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.322501 959 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:55.322516 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4b90205ef15f470a99c013e7f2cf99c6 (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]). Expected version 3 got 0
I20250115 14:51:55.323472 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d51e53b4ffa45bfbb9553d7a5870844 with delete_type TABLET_DATA_DELETED (replaced by tablet b29ef998d8dc4d4b9bb0a824a850d824) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.325052 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aeca9975f1945ad956f47db1ea16a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.327404 879 tablet_service.cc:1467] Processing CreateTablet for tablet caed629878594954842dfc6a40cc8b6d (DEFAULT_TABLE table=default.Table_b1b29136da42404eb226ca785d5aa7bc [id=b6aeb933c5e64e88a6d794a98650acbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.327505 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 496dec75500347699625a755dceed2bb (DEFAULT_TABLE table=default.Table_cb0308c238d7479dbcc041fe3d6e7eff [id=983e866df38543b0b2502165df44b94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.327854 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caed629878594954842dfc6a40cc8b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.327863 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 496dec75500347699625a755dceed2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.330668 1162 tablet_bootstrap.cc:492] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.330825 1162 ts_tablet_manager.cc:1397] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20250115 14:51:55.331629 1162 raft_consensus.cc:357] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.331868 1162 raft_consensus.cc:383] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.331949 1162 raft_consensus.cc:738] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.332733 888 tablet_service.cc:1467] Processing CreateTablet for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (DEFAULT_TABLE table=default.Table_3c51ba0db5274d3cafe5c89ecf27a771 [id=be99ec07535243a68c1b3ee1be0c62e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.332782 1162 consensus_queue.cc:260] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.333098 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.335317 1022 tablet_service.cc:1467] Processing CreateTablet for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (DEFAULT_TABLE table=default.Table_3c51ba0db5274d3cafe5c89ecf27a771 [id=be99ec07535243a68c1b3ee1be0c62e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.335740 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.336681 1162 ts_tablet_manager.cc:1428] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250115 14:51:55.340267 1162 tablet_bootstrap.cc:492] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.340685 746 tablet_service.cc:1467] Processing CreateTablet for tablet 496dec75500347699625a755dceed2bb (DEFAULT_TABLE table=default.Table_cb0308c238d7479dbcc041fe3d6e7eff [id=983e866df38543b0b2502165df44b94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.341192 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 496dec75500347699625a755dceed2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.341791 1162 tablet_bootstrap.cc:654] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.344548 754 tablet_service.cc:1467] Processing CreateTablet for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (DEFAULT_TABLE table=default.Table_3c51ba0db5274d3cafe5c89ecf27a771 [id=be99ec07535243a68c1b3ee1be0c62e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.346251 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.347986 1162 tablet_bootstrap.cc:492] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.348131 1162 ts_tablet_manager.cc:1397] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20250115 14:51:55.348898 1162 raft_consensus.cc:357] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.349107 1162 raft_consensus.cc:383] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.349817 1162 raft_consensus.cc:738] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.349763 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d5e0fb344ffa468286ab7ab95c01b841" } new_table_name: "default.Table_5b812a62d552484794327ad5be4f5286" modify_external_catalogs: true
I20250115 14:51:55.350093 1162 consensus_queue.cc:260] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.351172 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b5c4f807878b4872bb3a73a6f9532957"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.351697 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5c4f807878b4872bb3a73a6f9532957 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.353073 775 raft_consensus.cc:3054] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.356526 1168 ts_tablet_manager.cc:1428] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20250115 14:51:55.359252 1162 ts_tablet_manager.cc:1428] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250115 14:51:55.363000 1162 tablet_bootstrap.cc:492] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.363175 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_02593da1cd7245f38f85424d1a4baad9" } modify_external_catalogs: true
I20250115 14:51:55.363370 1168 tablet_bootstrap.cc:492] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.363371 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_02593da1cd7245f38f85424d1a4baad9" } modify_external_catalogs: true
I20250115 14:51:55.364878 1164 tablet_bootstrap.cc:492] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:55.365123 1164 ts_tablet_manager.cc:1397] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.006s
I20250115 14:51:55.365401 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d3300e90dc3845f78b3c63dfbf681dde" } modify_external_catalogs: true
I20250115 14:51:55.365525 1225 raft_consensus.cc:491] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.365586 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d3300e90dc3845f78b3c63dfbf681dde" } modify_external_catalogs: true
I20250115 14:51:55.365658 1225 raft_consensus.cc:513] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.365916 1164 raft_consensus.cc:357] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.366127 1164 raft_consensus.cc:383] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.366197 1164 raft_consensus.cc:738] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.366235 1168 tablet_bootstrap.cc:654] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.366326 1225 leader_election.cc:290] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.366426 1164 consensus_queue.cc:260] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.366973 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae60842a2d4486eb7651c3f710ee1e8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:55.367125 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c75173371e554bdb8d99dcf3cea7c244" } new_table_name: "default.Table_1699f914f5e94be1b121f649a762e07b" modify_external_catalogs: true
I20250115 14:51:55.367197 909 raft_consensus.cc:2463] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:55.367408 1164 ts_tablet_manager.cc:1428] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:55.367683 1164 tablet_bootstrap.cc:492] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:55.369383 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fb7d4709dd544378b8802e3294d6e441" } modify_external_catalogs: true
I20250115 14:51:55.369560 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fb7d4709dd544378b8802e3294d6e441" } modify_external_catalogs: true
I20250115 14:51:55.369767 1164 tablet_bootstrap.cc:654] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.370128 983 leader_election.cc:304] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:55.371018 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae60842a2d4486eb7651c3f710ee1e8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:55.371563 984 leader_election.cc:343] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.372571 1225 raft_consensus.cc:2798] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:55.372695 1225 raft_consensus.cc:491] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:55.372784 1225 raft_consensus.cc:3054] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.375317 1225 raft_consensus.cc:513] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.375670 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b92231509c754e8dbfc564813a76a123
I20250115 14:51:55.376142 1231 raft_consensus.cc:491] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.376269 1231 raft_consensus.cc:513] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:55.376511 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a915e0c0a30940ed8a6233f117e2a560 consensus state has no leader
I20250115 14:51:55.376885 1231 leader_election.cc:290] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.377453 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b92231509c754e8dbfc564813a76a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.377445 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b90205ef15f470a99c013e7f2cf99c6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:55.377753 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b90205ef15f470a99c013e7f2cf99c6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:55.377863 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae60842a2d4486eb7651c3f710ee1e8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:55.378286 983 leader_election.cc:343] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.378849 888 tablet_service.cc:1514] Processing DeleteTablet for tablet b92231509c754e8dbfc564813a76a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.383669 1209 raft_consensus.cc:491] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.383838 1209 raft_consensus.cc:513] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.384521 1209 leader_election.cc:290] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:55.384842 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_620e744402d54372bb5c38ad4e168dc1" } new_table_name: "default.Table_50d67783a2d6443097b0027856df1225" modify_external_catalogs: true
W20250115 14:51:55.385566 984 leader_election.cc:343] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.385682 984 leader_election.cc:304] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.385993 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_12bf6210790a443dbd238713e2f8a874" } new_table_name: "default.Table_6e6412160f1746a3a9f165ebe103ee59" modify_external_catalogs: true
W20250115 14:51:55.386093 984 leader_election.cc:343] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.387490 1167 ts_tablet_manager.cc:1918] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.387629 1167 log.cc:1198] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/a6b76cb343fb4456ac07fd3c6e9888de
I20250115 14:51:55.387681 1231 raft_consensus.cc:2743] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:55.387979 1167 ts_tablet_manager.cc:1939] T a6b76cb343fb4456ac07fd3c6e9888de P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.387982 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7580b6245b41628874d9dcbc3366e2" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:55.388558 847 leader_election.cc:343] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.388983 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7580b6245b41628874d9dcbc3366e2" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:55.389492 848 leader_election.cc:343] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.389614 848 leader_election.cc:304] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.389954 1209 raft_consensus.cc:2743] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:55.387346 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a1bd89ddf322401092a05b3f20970798"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.390496 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 9bcfab1e1f0349739eff636fb1b04806 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.391081 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b17b7ebacc9a4d32b756eff66de640dd" } new_table_name: "default.Table_478cc36362494028a43f0e71465e1834" modify_external_catalogs: true
W20250115 14:51:55.396643 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1bd89ddf322401092a05b3f20970798 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.398087 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 00adfe53c726428b9ac2fcb47756b3de
I20250115 14:51:55.398451 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a4fd4708104e4d52971393e77f803df3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.398983 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4fd4708104e4d52971393e77f803df3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:55.400588 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7233ebf3bae1462d8eeb3669b39944c0 consensus state has no leader
I20250115 14:51:55.400985 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a9098644161a45d4be352eb0b77e7eca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.401436 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9098644161a45d4be352eb0b77e7eca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.403229 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet fad6f9f7f853418e85353edfd38cc80e
I20250115 14:51:55.404377 447 master_service.cc:946] ReplaceTablet: received request to replace tablet ac57fa232ae4462793b021b60ede0973 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.405537 888 tablet_service.cc:1514] Processing DeleteTablet for tablet fad6f9f7f853418e85353edfd38cc80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.405905 909 raft_consensus.cc:1270] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:55.405928 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet fad6f9f7f853418e85353edfd38cc80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.406210 751 tablet_service.cc:1514] Processing DeleteTablet for tablet fad6f9f7f853418e85353edfd38cc80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.406450 1231 consensus_queue.cc:1035] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:51:55.406646 984 consensus_peers.cc:487] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:55.408838 1225 leader_election.cc:290] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.412506 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet a6b76cb343fb4456ac07fd3c6e9888de (table default.Table_91244a7b0fa14291bcc2e9a2d776b4ae [id=4a31d55ea43e4ce5bbda5b78fdbfffa6]) successfully deleted
W20250115 14:51:55.420888 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f847f92cd122423aa93f1afa701b0bbb consensus state has no leader
I20250115 14:51:55.420886 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet b92231509c754e8dbfc564813a76a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.421188 1521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef770dae727b42f5a9513b6f3d68b84b. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:55.421430 1521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:51:55.421942 449 master_service.cc:946] ReplaceTablet: received request to replace tablet e932d5b820f94130967eccce9e1542fa from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.349462 451 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]). Expected version 2 got 0
W20250115 14:51:55.427062 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e990a2920b542f69a8bdd022db6e7e2 consensus state has no leader
I20250115 14:51:55.433226 888 tablet_service.cc:1467] Processing CreateTablet for tablet 496dec75500347699625a755dceed2bb (DEFAULT_TABLE table=default.Table_cb0308c238d7479dbcc041fe3d6e7eff [id=983e866df38543b0b2502165df44b94c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.437872 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5750e04cfa8d489286be4eceb1125abe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.438534 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6dfc58433fe64bf48f2215d6caab2368" } modify_external_catalogs: true
I20250115 14:51:55.438786 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6dfc58433fe64bf48f2215d6caab2368" } modify_external_catalogs: true
I20250115 14:51:55.442070 1146 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:55.442353 1146 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: This master's current role is: LEADER
I20250115 14:51:55.454236 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_872859e534b64345b7989072c4d38b8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.455454 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d76649a3901e45f4826621907456750f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.456383 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bbb6df0135694d7b83ed96983e487001"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.458818 1162 tablet_bootstrap.cc:654] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.462782 751 tablet_service.cc:1467] Processing CreateTablet for tablet e198b4e1a65b4704bf2bb8f7d5987385 (DEFAULT_TABLE table=default.Table_9c06e47841fa45c59c310505e26c9361 [id=a3bf28673dac4ddab579709d6fd8885d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.463250 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e198b4e1a65b4704bf2bb8f7d5987385. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.463289 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a43e4255e5c3414aa47bff8bbe111d13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.465307 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5750e04cfa8d489286be4eceb1125abe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:55.465458 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d76649a3901e45f4826621907456750f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:55.465652 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbb6df0135694d7b83ed96983e487001 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.466696 1209 raft_consensus.cc:491] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.466848 1209 raft_consensus.cc:513] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.467490 1209 leader_election.cc:290] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:55.468647 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745153c106844076bb020f9ed026c97a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:55.469678 847 leader_election.cc:343] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:55.470265 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_872859e534b64345b7989072c4d38b8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.470659 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745153c106844076bb020f9ed026c97a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:55.471242 848 leader_election.cc:343] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.471390 848 leader_election.cc:304] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.471809 1209 raft_consensus.cc:2743] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:55.477411 1166 tablet_replica.cc:331] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:55.477699 1166 raft_consensus.cc:2238] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.477953 1166 raft_consensus.cc:2267] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.478621 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4821224ac8f1472d8b1e8afc06a118a3 (table default.Table_721928c380b840ad84f9f5030b52632f [id=c5f831177088497dbf1fee130e231e37]). Expected version 1 got 0
W20250115 14:51:55.480026 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5027fba8fed34db59b484918256abc45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5027fba8fed34db59b484918256abc45 already in progress: creating tablet
W20250115 14:51:55.480321 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f55c75e5febb407398b0bc06a1db9b1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f55c75e5febb407398b0bc06a1db9b1a already in progress: creating tablet
I20250115 14:51:55.485400 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e932d5b820f94130967eccce9e1542fa
I20250115 14:51:55.485875 449 catalog_manager.cc:6596] ReplaceTablet: tablet e932d5b820f94130967eccce9e1542fa deleted and replaced by tablet fc3d2ca7cd7c48a79ed1396fbbdc6866
I20250115 14:51:55.487443 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4821224ac8f1472d8b1e8afc06a118a3 (table default.Table_721928c380b840ad84f9f5030b52632f [id=c5f831177088497dbf1fee130e231e37]). Expected version 1 got 0
I20250115 14:51:55.488301 879 tablet_service.cc:1514] Processing DeleteTablet for tablet e932d5b820f94130967eccce9e1542fa with delete_type TABLET_DATA_DELETED (replaced by tablet fc3d2ca7cd7c48a79ed1396fbbdc6866) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.488879 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e932d5b820f94130967eccce9e1542fa with delete_type TABLET_DATA_DELETED (replaced by tablet fc3d2ca7cd7c48a79ed1396fbbdc6866) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.490489 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ac57fa232ae4462793b021b60ede0973
I20250115 14:51:55.490978 447 catalog_manager.cc:6596] ReplaceTablet: tablet ac57fa232ae4462793b021b60ede0973 deleted and replaced by tablet cea16f889c73400a85b9553c1975457e
I20250115 14:51:55.491639 879 tablet_service.cc:1514] Processing DeleteTablet for tablet ac57fa232ae4462793b021b60ede0973 with delete_type TABLET_DATA_DELETED (replaced by tablet cea16f889c73400a85b9553c1975457e) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.491859 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet ac57fa232ae4462793b021b60ede0973 with delete_type TABLET_DATA_DELETED (replaced by tablet cea16f889c73400a85b9553c1975457e) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.492121 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ac57fa232ae4462793b021b60ede0973 with delete_type TABLET_DATA_DELETED (replaced by tablet cea16f889c73400a85b9553c1975457e) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:55.493454 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a73bd0141ccc4ec4b052b18b9fb62bb7 consensus state has no leader
I20250115 14:51:55.493891 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e50462626f914fb1b0f440c48f30e72d
I20250115 14:51:55.494158 1162 tablet_bootstrap.cc:492] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.494328 1162 ts_tablet_manager.cc:1397] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.131s user 0.003s sys 0.001s
I20250115 14:51:55.495128 1162 raft_consensus.cc:357] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.495365 1162 raft_consensus.cc:383] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.495389 879 tablet_service.cc:1514] Processing DeleteTablet for tablet e50462626f914fb1b0f440c48f30e72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.495442 1162 raft_consensus.cc:738] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.495692 1162 consensus_queue.cc:260] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.495946 1209 consensus_queue.cc:1035] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:55.496479 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet e50462626f914fb1b0f440c48f30e72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.496884 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e50462626f914fb1b0f440c48f30e72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.498091 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet e932d5b820f94130967eccce9e1542fa with delete_type TABLET_DATA_DELETED (replaced by tablet fc3d2ca7cd7c48a79ed1396fbbdc6866) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.498333 1162 ts_tablet_manager.cc:1428] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:51:55.498631 1162 tablet_bootstrap.cc:492] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.500377 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4e0b412e80014674bc22990db5953cc6" } modify_external_catalogs: true
I20250115 14:51:55.501211 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ae60842a2d4486eb7651c3f710ee1e8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:55.501434 910 raft_consensus.cc:3054] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.504261 910 raft_consensus.cc:2463] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:55.504940 983 leader_election.cc:304] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.506747 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 496dec75500347699625a755dceed2bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.507474 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d243daba84ff4befa0c4e7bac5fe7476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.507860 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4e0b412e80014674bc22990db5953cc6" } modify_external_catalogs: true
I20250115 14:51:55.510011 1231 raft_consensus.cc:2798] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:55.510780 1023 tablet_service.cc:1467] Processing CreateTablet for tablet e198b4e1a65b4704bf2bb8f7d5987385 (DEFAULT_TABLE table=default.Table_9c06e47841fa45c59c310505e26c9361 [id=a3bf28673dac4ddab579709d6fd8885d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.510931 879 tablet_service.cc:1467] Processing CreateTablet for tablet e198b4e1a65b4704bf2bb8f7d5987385 (DEFAULT_TABLE table=default.Table_9c06e47841fa45c59c310505e26c9361 [id=a3bf28673dac4ddab579709d6fd8885d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.511230 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e198b4e1a65b4704bf2bb8f7d5987385. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.511318 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e198b4e1a65b4704bf2bb8f7d5987385. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.513537 1231 raft_consensus.cc:695] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:55.513830 1231 consensus_queue.cc:237] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:55.515235 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a43e4255e5c3414aa47bff8bbe111d13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:55.517796 848 consensus_peers.cc:487] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:55.518770 1016 tablet_service.cc:1467] Processing CreateTablet for tablet bb19d39258e444799d403a43475d9c8a (DEFAULT_TABLE table=default.Table_e6612d24c40f44a780551d89e880543a [id=13c49e5a84b94acca8344f3e04e99736]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.519217 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb19d39258e444799d403a43475d9c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.522308 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_124bdb85e4194ff5a214a6b79870a1cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.525477 879 tablet_service.cc:1467] Processing CreateTablet for tablet bb19d39258e444799d403a43475d9c8a (DEFAULT_TABLE table=default.Table_e6612d24c40f44a780551d89e880543a [id=13c49e5a84b94acca8344f3e04e99736]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.525609 1166 ts_tablet_manager.cc:1905] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.525908 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb19d39258e444799d403a43475d9c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.529484 751 tablet_service.cc:1467] Processing CreateTablet for tablet bb19d39258e444799d403a43475d9c8a (DEFAULT_TABLE table=default.Table_e6612d24c40f44a780551d89e880543a [id=13c49e5a84b94acca8344f3e04e99736]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.529915 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb19d39258e444799d403a43475d9c8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.531512 740 tablet_service.cc:1467] Processing CreateTablet for tablet 171643c40c9b43ffb823f7c33789284d (DEFAULT_TABLE table=default.Table_d98a5978bcea4e01af0c694f42d2308a [id=7628dbb6818f443680a97fef665d6d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.532059 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171643c40c9b43ffb823f7c33789284d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:55.540047 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 276abefcab6a422aa6467f00864c01f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 276abefcab6a422aa6467f00864c01f4 already in progress: creating tablet
I20250115 14:51:55.542332 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]). Expected version 2 got 0
W20250115 14:51:55.542609 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 5027fba8fed34db59b484918256abc45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5027fba8fed34db59b484918256abc45 already in progress: creating tablet
I20250115 14:51:55.543138 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 79aad920dd9744d9b87a8234c92232d8
W20250115 14:51:55.544190 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet bd4102b1f83e414d93a6a68315e36d16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4102b1f83e414d93a6a68315e36d16 already in progress: creating tablet
I20250115 14:51:55.546329 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aad920dd9744d9b87a8234c92232d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:55.548624 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet bc8337d181d44ac7bb82551a677b6831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8337d181d44ac7bb82551a677b6831 already in progress: creating tablet
W20250115 14:51:55.549399 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e3fbde011216426cb41c51fe359dcb5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3fbde011216426cb41c51fe359dcb5f already in progress: creating tablet
I20250115 14:51:55.552672 1167 tablet_replica.cc:331] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.552883 1167 raft_consensus.cc:2238] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.553112 1167 raft_consensus.cc:2267] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.553664 1167 ts_tablet_manager.cc:1905] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.554832 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 2711a7d437f44b2fa11504d8e400e360 (table default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8 [id=a02ee91f154e41fd9ee54b476fd77331]). Expected version 1 got 0
W20250115 14:51:55.557734 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_124bdb85e4194ff5a214a6b79870a1cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.557720 888 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aad920dd9744d9b87a8234c92232d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.561161 751 tablet_service.cc:1467] Processing CreateTablet for tablet d5ffa997ca6c46559830b1dedf2d1a66 (DEFAULT_TABLE table=default.Table_90f2e2c555f84fe39b5b478581ef0e44 [id=2c2d7017c98f49d09e57327f21412e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.562218 888 tablet_service.cc:1467] Processing CreateTablet for tablet 171643c40c9b43ffb823f7c33789284d (DEFAULT_TABLE table=default.Table_d98a5978bcea4e01af0c694f42d2308a [id=7628dbb6818f443680a97fef665d6d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.562346 878 tablet_service.cc:1467] Processing CreateTablet for tablet d5ffa997ca6c46559830b1dedf2d1a66 (DEFAULT_TABLE table=default.Table_90f2e2c555f84fe39b5b478581ef0e44 [id=2c2d7017c98f49d09e57327f21412e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.562737 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ffa997ca6c46559830b1dedf2d1a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.562994 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171643c40c9b43ffb823f7c33789284d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.563545 752 tablet_service.cc:1467] Processing CreateTablet for tablet d5dd41d42331494e9c1b021816d0287e (DEFAULT_TABLE table=default.Table_ae4cccbf33234ae28a199f7926716b28 [id=1ab58cc2377b41589f5df2505f6b1208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.571159 1168 tablet_bootstrap.cc:492] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.571323 1168 ts_tablet_manager.cc:1397] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.208s user 0.019s sys 0.023s
I20250115 14:51:55.572207 1168 raft_consensus.cc:357] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.572444 1168 raft_consensus.cc:383] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.572530 1168 raft_consensus.cc:738] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.572782 1168 consensus_queue.cc:260] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.573455 1168 ts_tablet_manager.cc:1428] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:55.573745 1168 tablet_bootstrap.cc:492] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.581650 1168 tablet_bootstrap.cc:654] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.590657 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_67aaf987cc30461bb07c319c25639ffa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.591416 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67aaf987cc30461bb07c319c25639ffa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.592522 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_435f7dd182634613bd6116abea02ff0f" } new_table_name: "default.Table_fc422b5daf4a4770945aab3d34bc5d35" modify_external_catalogs: true
I20250115 14:51:55.594260 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d7378da7b662410d864c634dd5e34926" } modify_external_catalogs: true
I20250115 14:51:55.594575 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d7378da7b662410d864c634dd5e34926" } modify_external_catalogs: true
I20250115 14:51:55.599076 1168 tablet_bootstrap.cc:492] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.599274 1168 ts_tablet_manager.cc:1397] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20250115 14:51:55.600073 1168 raft_consensus.cc:357] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.600317 1168 raft_consensus.cc:383] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.600383 1168 raft_consensus.cc:738] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.600630 1168 consensus_queue.cc:260] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.605072 886 tablet_service.cc:1467] Processing CreateTablet for tablet dff155a5c2b0479a90e74993de6f402d (DEFAULT_TABLE table=default.Table_8c90e9dde49d43239bcb743145f4ef07 [id=f6da20956eb349709bbe9e43a41fe4ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.605465 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff155a5c2b0479a90e74993de6f402d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.605423 890 tablet_service.cc:1467] Processing CreateTablet for tablet d195be2d543f4a8693045a024912c43a (DEFAULT_TABLE table=default.Table_bba588d115d34226951843abdb9b6ebe [id=e99d33d2241343e2818d4217babf9a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.605757 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d195be2d543f4a8693045a024912c43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.608556 881 tablet_service.cc:1467] Processing CreateTablet for tablet 59d8fd4b26294e8ca6745b95f2e03231 (DEFAULT_TABLE table=default.Table_27c15993e89541219ace5c7251953112 [id=757cda13648a4e3d9b02d70555966106]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.608968 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59d8fd4b26294e8ca6745b95f2e03231. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.610433 743 tablet_service.cc:1467] Processing CreateTablet for tablet 37ad2b7350294375bd30fb4b46906491 (DEFAULT_TABLE table=default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.611040 741 tablet_service.cc:1467] Processing CreateTablet for tablet 59d8fd4b26294e8ca6745b95f2e03231 (DEFAULT_TABLE table=default.Table_27c15993e89541219ace5c7251953112 [id=757cda13648a4e3d9b02d70555966106]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.611646 756 tablet_service.cc:1467] Processing CreateTablet for tablet dff155a5c2b0479a90e74993de6f402d (DEFAULT_TABLE table=default.Table_8c90e9dde49d43239bcb743145f4ef07 [id=f6da20956eb349709bbe9e43a41fe4ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.613058 744 tablet_service.cc:1467] Processing CreateTablet for tablet d195be2d543f4a8693045a024912c43a (DEFAULT_TABLE table=default.Table_bba588d115d34226951843abdb9b6ebe [id=e99d33d2241343e2818d4217babf9a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.613144 878 tablet_service.cc:1467] Processing CreateTablet for tablet d5dd41d42331494e9c1b021816d0287e (DEFAULT_TABLE table=default.Table_ae4cccbf33234ae28a199f7926716b28 [id=1ab58cc2377b41589f5df2505f6b1208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.613466 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5dd41d42331494e9c1b021816d0287e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.613644 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d3b142be26bc4ef59f6c07abad93e201" } modify_external_catalogs: true
I20250115 14:51:55.613932 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d3b142be26bc4ef59f6c07abad93e201" } modify_external_catalogs: true
I20250115 14:51:55.556396 1162 tablet_bootstrap.cc:654] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.562553 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 171643c40c9b43ffb823f7c33789284d (DEFAULT_TABLE table=default.Table_d98a5978bcea4e01af0c694f42d2308a [id=7628dbb6818f443680a97fef665d6d2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.615545 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0ed87b7187f342c8904d2f6d559034e0" } new_table_name: "default.Table_875d7bd9588344008279ca89c1ad52be" modify_external_catalogs: true
I20250115 14:51:55.562605 1015 tablet_service.cc:1467] Processing CreateTablet for tablet d5ffa997ca6c46559830b1dedf2d1a66 (DEFAULT_TABLE table=default.Table_90f2e2c555f84fe39b5b478581ef0e44 [id=2c2d7017c98f49d09e57327f21412e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.574415 1540 raft_consensus.cc:491] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.588366 1231 consensus_queue.cc:1035] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20250115 14:51:55.616133 1540 raft_consensus.cc:513] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.604198 1027 tablet_service.cc:1467] Processing CreateTablet for tablet d195be2d543f4a8693045a024912c43a (DEFAULT_TABLE table=default.Table_bba588d115d34226951843abdb9b6ebe [id=e99d33d2241343e2818d4217babf9a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.604727 1026 tablet_service.cc:1467] Processing CreateTablet for tablet dff155a5c2b0479a90e74993de6f402d (DEFAULT_TABLE table=default.Table_8c90e9dde49d43239bcb743145f4ef07 [id=f6da20956eb349709bbe9e43a41fe4ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.616811 1540 leader_election.cc:290] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.605248 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 59d8fd4b26294e8ca6745b95f2e03231 (DEFAULT_TABLE table=default.Table_27c15993e89541219ace5c7251953112 [id=757cda13648a4e3d9b02d70555966106]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.556816 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aad920dd9744d9b87a8234c92232d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.562850 1014 tablet_service.cc:1467] Processing CreateTablet for tablet d5dd41d42331494e9c1b021816d0287e (DEFAULT_TABLE table=default.Table_ae4cccbf33234ae28a199f7926716b28 [id=1ab58cc2377b41589f5df2505f6b1208]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.618098 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d243daba84ff4befa0c4e7bac5fe7476" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:55.618304 913 raft_consensus.cc:2463] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:55.618492 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d243daba84ff4befa0c4e7bac5fe7476" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:55.619117 984 leader_election.cc:343] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.619753 983 leader_election.cc:304] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.620057 1540 raft_consensus.cc:2798] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:55.620151 1540 raft_consensus.cc:491] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:55.620221 1540 raft_consensus.cc:3054] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.622102 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5750e04cfa8d489286be4eceb1125abe" } modify_external_catalogs: true
I20250115 14:51:55.622573 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 50ca710cea154be5b0f57ace4af4168b from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.622732 1540 raft_consensus.cc:513] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.623364 1540 leader_election.cc:290] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.623831 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d243daba84ff4befa0c4e7bac5fe7476" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:55.623854 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d243daba84ff4befa0c4e7bac5fe7476" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:55.624048 913 raft_consensus.cc:3054] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:55.624424 984 leader_election.cc:343] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.626792 913 raft_consensus.cc:2463] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:55.627223 983 leader_election.cc:304] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.627521 1540 raft_consensus.cc:2798] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:55.627662 1540 raft_consensus.cc:695] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:55.629204 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_881f3ef58e824b4cb4e1fb3ec4846cc7" } new_table_name: "default.Table_13d7e84841db4e7897fa09136b16e197" modify_external_catalogs: true
W20250115 14:51:55.630163 984 consensus_peers.cc:487] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:55.631695 1540 consensus_queue.cc:237] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.637684 1168 ts_tablet_manager.cc:1428] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.038s user 0.000s sys 0.002s
I20250115 14:51:55.638031 1168 tablet_bootstrap.cc:492] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.638357 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 0291acf45d75421b8b1c6311560e7f0c from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.646857 1168 tablet_bootstrap.cc:654] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.657337 1162 tablet_bootstrap.cc:492] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.657618 1162 ts_tablet_manager.cc:1397] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.159s user 0.004s sys 0.001s
I20250115 14:51:55.658983 1162 raft_consensus.cc:357] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.659397 1162 raft_consensus.cc:383] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.659631 1162 raft_consensus.cc:738] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.660153 1162 consensus_queue.cc:260] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.661526 1162 ts_tablet_manager.cc:1428] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250115 14:51:55.662215 1557 tablet.cc:1721] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:51:55.667860 984 consensus_peers.cc:487] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:51:55.668687 984 consensus_peers.cc:487] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:55.679916 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5dd41d42331494e9c1b021816d0287e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.684639 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 171643c40c9b43ffb823f7c33789284d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.687105 1168 tablet_bootstrap.cc:492] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.687252 1168 ts_tablet_manager.cc:1397] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.004s
I20250115 14:51:55.690201 1168 raft_consensus.cc:357] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.690443 1168 raft_consensus.cc:383] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.691329 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d195be2d543f4a8693045a024912c43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.691501 1168 raft_consensus.cc:738] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.691774 1168 consensus_queue.cc:260] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.696566 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ffa997ca6c46559830b1dedf2d1a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.696626 1162 tablet_bootstrap.cc:492] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.698333 1162 tablet_bootstrap.cc:654] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.702200 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59d8fd4b26294e8ca6745b95f2e03231. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.704530 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff155a5c2b0479a90e74993de6f402d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.710158 1168 ts_tablet_manager.cc:1428] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250115 14:51:55.712616 1167 ts_tablet_manager.cc:1918] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.712781 1167 log.cc:1198] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f55c75e5febb407398b0bc06a1db9b1a
I20250115 14:51:55.719851 1517 tablet.cc:1721] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:55.720458 1168 tablet_bootstrap.cc:492] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.720484 1167 ts_tablet_manager.cc:1939] T f55c75e5febb407398b0bc06a1db9b1a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.721494 1231 raft_consensus.cc:491] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.721662 1231 raft_consensus.cc:513] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.722360 1231 leader_election.cc:290] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.722975 1162 tablet_bootstrap.cc:492] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.723138 1162 ts_tablet_manager.cc:1397] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250115 14:51:55.723996 1162 raft_consensus.cc:357] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.724226 1162 raft_consensus.cc:383] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.724318 1162 raft_consensus.cc:738] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.724812 1162 consensus_queue.cc:260] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.726086 1162 ts_tablet_manager.cc:1428] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250115 14:51:55.727770 1162 tablet_bootstrap.cc:492] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.728899 1168 tablet_bootstrap.cc:654] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.731395 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745153c106844076bb020f9ed026c97a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:55.731988 984 leader_election.cc:343] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.733484 1167 tablet_replica.cc:331] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.733518 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745153c106844076bb020f9ed026c97a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:55.733705 913 raft_consensus.cc:2463] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:55.734133 983 leader_election.cc:304] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.733701 1167 raft_consensus.cc:2238] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.734952 1167 raft_consensus.cc:2267] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.734984 1231 raft_consensus.cc:2798] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:55.735088 1231 raft_consensus.cc:491] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:55.735456 1167 ts_tablet_manager.cc:1905] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.735666 1231 raft_consensus.cc:3054] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.736369 1549 raft_consensus.cc:491] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.739068 1231 raft_consensus.cc:513] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.739251 1549 raft_consensus.cc:513] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.740458 1561 tablet.cc:1721] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:55.740476 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b76cb343fb4456ac07fd3c6e9888de" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:55.741997 1549 leader_election.cc:290] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:55.742826 984 leader_election.cc:343] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:55.743029 983 leader_election.cc:343] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: a6b76cb343fb4456ac07fd3c6e9888de
I20250115 14:51:55.743310 983 leader_election.cc:304] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.744246 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745153c106844076bb020f9ed026c97a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:55.744870 984 leader_election.cc:343] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.745097 1549 raft_consensus.cc:2743] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:55.745424 1231 leader_election.cc:290] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.746110 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745153c106844076bb020f9ed026c97a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:55.746300 913 raft_consensus.cc:3054] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.747754 1167 ts_tablet_manager.cc:1918] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.747890 1167 log.cc:1198] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d243daba84ff4befa0c4e7bac5fe7476
I20250115 14:51:55.749765 913 raft_consensus.cc:2463] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:55.750198 983 leader_election.cc:304] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.748253 1167 ts_tablet_manager.cc:1939] T d243daba84ff4befa0c4e7bac5fe7476 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.753551 1549 raft_consensus.cc:491] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.753710 1549 raft_consensus.cc:513] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.755156 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3ec000089a47a2930dec382192cd6f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:55.756943 1549 leader_election.cc:290] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.757755 1231 raft_consensus.cc:2798] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:51:55.759665 984 leader_election.cc:343] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.760380 1168 tablet_bootstrap.cc:492] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.760548 1168 ts_tablet_manager.cc:1397] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.005s
I20250115 14:51:55.761116 1231 raft_consensus.cc:695] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:55.761288 1168 raft_consensus.cc:357] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.761521 1168 raft_consensus.cc:383] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.761595 1168 raft_consensus.cc:738] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.761834 1168 consensus_queue.cc:260] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.762447 1168 ts_tablet_manager.cc:1428] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:55.762610 1231 consensus_queue.cc:237] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.771571 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3ec000089a47a2930dec382192cd6f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:55.771900 913 raft_consensus.cc:2463] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:55.775002 983 leader_election.cc:304] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.775295 1549 raft_consensus.cc:2798] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:55.775408 1549 raft_consensus.cc:491] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:55.775490 1549 raft_consensus.cc:3054] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.778167 1549 raft_consensus.cc:513] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.778764 959 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:51:55.779306 1549 leader_election.cc:290] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.780027 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3ec000089a47a2930dec382192cd6f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:55.780305 913 raft_consensus.cc:3054] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.782616 1162 tablet_bootstrap.cc:654] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.781636 1167 tablet_replica.cc:331] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.781873 1168 tablet_bootstrap.cc:492] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.786509 1168 tablet_bootstrap.cc:654] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.787091 1167 raft_consensus.cc:2238] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.787286 1167 raft_consensus.cc:2267] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.787868 1167 ts_tablet_manager.cc:1905] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.789475 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bb792ca1b3f54def92f504ec8682d2c7 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]). Expected version 2 got 0
I20250115 14:51:55.792691 1561 tablet.cc:1721] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:55.793239 1517 tablet.cc:1721] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:55.783233 913 raft_consensus.cc:2463] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:55.800329 983 leader_election.cc:304] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:55.801343 1173 raft_consensus.cc:491] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.801525 1173 raft_consensus.cc:513] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.802189 1173 leader_election.cc:290] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:55.802371 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3ec000089a47a2930dec382192cd6f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:55.802876 1209 raft_consensus.cc:491] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:51:55.802887 984 leader_election.cc:343] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.803049 1209 raft_consensus.cc:513] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:55.803329 710 leader_election.cc:343] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 13c00f03edc74af8b084bb236487a347
I20250115 14:51:55.803663 1549 raft_consensus.cc:2798] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:55.803691 1209 leader_election.cc:290] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:55.803886 1549 raft_consensus.cc:695] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:55.804185 1549 consensus_queue.cc:237] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.804378 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5027fba8fed34db59b484918256abc45" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:55.804874 847 leader_election.cc:343] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.805335 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5027fba8fed34db59b484918256abc45" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:55.805876 848 leader_election.cc:343] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.805987 848 leader_election.cc:304] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.806358 1209 raft_consensus.cc:2743] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:55.809103 1168 tablet_bootstrap.cc:492] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.809240 1168 ts_tablet_manager.cc:1397] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.000s
I20250115 14:51:55.809842 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ffa997ca6c46559830b1dedf2d1a66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.809996 1168 raft_consensus.cc:357] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.810185 1168 raft_consensus.cc:383] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.810261 1168 raft_consensus.cc:738] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.810334 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5dd41d42331494e9c1b021816d0287e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.810447 1168 consensus_queue.cc:260] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.810689 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37ad2b7350294375bd30fb4b46906491. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.811048 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59d8fd4b26294e8ca6745b95f2e03231. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.811383 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff155a5c2b0479a90e74993de6f402d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.811708 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d195be2d543f4a8693045a024912c43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.812283 1168 ts_tablet_manager.cc:1428] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:55.812556 1168 tablet_bootstrap.cc:492] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.814796 1168 tablet_bootstrap.cc:654] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:55.816478 710 leader_election.cc:343] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 13c00f03edc74af8b084bb236487a347
I20250115 14:51:55.816612 710 leader_election.cc:304] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:55.817252 1173 raft_consensus.cc:2743] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:55.819787 983 consensus_peers.cc:487] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 -> Peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Couldn't send request to peer 73c86158cc2641f182ea50a06634592d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fd72a0d586d4a2c817f6a39daa07063. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:55.824828 1168 tablet_bootstrap.cc:492] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.825014 1168 ts_tablet_manager.cc:1397] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20250115 14:51:55.825276 1167 ts_tablet_manager.cc:1918] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.825383 1167 log.cc:1198] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/3ae60842a2d4486eb7651c3f710ee1e8
I20250115 14:51:55.825802 1167 ts_tablet_manager.cc:1939] T 3ae60842a2d4486eb7651c3f710ee1e8 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.826462 1168 raft_consensus.cc:357] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.826670 1168 raft_consensus.cc:383] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.827759 1168 raft_consensus.cc:738] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.828017 1168 consensus_queue.cc:260] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.828274 1167 tablet_replica.cc:331] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.828523 1167 raft_consensus.cc:2238] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.830085 1168 ts_tablet_manager.cc:1428] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:51:55.830355 1168 tablet_bootstrap.cc:492] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.833915 1168 tablet_bootstrap.cc:654] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.835230 1167 raft_consensus.cc:2267] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.835770 1167 ts_tablet_manager.cc:1905] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.839295 1162 tablet_bootstrap.cc:492] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.839432 1162 ts_tablet_manager.cc:1397] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.003s
I20250115 14:51:55.840137 1162 raft_consensus.cc:357] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.840366 1162 raft_consensus.cc:383] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.840449 1162 raft_consensus.cc:738] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.840687 1162 consensus_queue.cc:260] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.841214 1162 ts_tablet_manager.cc:1428] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:55.841466 1162 tablet_bootstrap.cc:492] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.842358 1168 tablet_bootstrap.cc:492] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.842490 1168 ts_tablet_manager.cc:1397] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20250115 14:51:55.843197 1168 raft_consensus.cc:357] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.845350 1168 raft_consensus.cc:383] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.846503 1162 tablet_bootstrap.cc:654] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.846537 1168 raft_consensus.cc:738] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.846808 1168 consensus_queue.cc:260] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.849413 1168 ts_tablet_manager.cc:1428] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250115 14:51:55.849676 1168 tablet_bootstrap.cc:492] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.851291 1168 tablet_bootstrap.cc:654] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.855726 1166 ts_tablet_manager.cc:1918] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.789880 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0de6f6f014074c3d8775cb5e4121ccb0
I20250115 14:51:55.792626 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5750e04cfa8d489286be4eceb1125abe" } modify_external_catalogs: true
I20250115 14:51:55.793094 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 293b10c333ef4f4ab1b743356a31de53
I20250115 14:51:55.859800 1167 ts_tablet_manager.cc:1918] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.861651 1167 log.cc:1198] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/745153c106844076bb020f9ed026c97a
W20250115 14:51:55.798384 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 consensus state has no leader
I20250115 14:51:55.862025 1167 ts_tablet_manager.cc:1939] T 745153c106844076bb020f9ed026c97a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.803503 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f55c75e5febb407398b0bc06a1db9b1a (table default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]) successfully deleted
W20250115 14:51:55.862548 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 59025us
W20250115 14:51:55.862931 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d86075ae5b894d5986ac4bd538f4d42a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d86075ae5b894d5986ac4bd538f4d42a already in progress: creating tablet
I20250115 14:51:55.859855 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 2711a7d437f44b2fa11504d8e400e360 (table default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8 [id=a02ee91f154e41fd9ee54b476fd77331]). Expected version 1 got 0
I20250115 14:51:55.865784 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b3a6b57353484cdf8f2dfde2738ae0c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.866607 1504 thread.cc:635] auto-rebalancer (catalog manager) Time spent creating pthread: real 0.590s user 0.079s sys 0.181s
W20250115 14:51:55.866737 1504 thread.cc:601] auto-rebalancer (catalog manager) Time spent starting thread: real 0.590s user 0.079s sys 0.181s
I20250115 14:51:55.865738 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_79fecf866fd6405a9b7df222877fc2b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.866317 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3fc104bc2f494a76b0f04ab5f41510bb" } new_table_name: "default.Table_c088519ba27c457983f341363f8a109e" modify_external_catalogs: true
I20250115 14:51:55.866930 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]). Expected version 1 got 0
I20250115 14:51:55.870096 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_01bf4fd686f74e258977d847e2ee966e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.870431 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79fecf866fd6405a9b7df222877fc2b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:55.870556 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01bf4fd686f74e258977d847e2ee966e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.872644 1167 tablet_replica.cc:331] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.872862 1167 raft_consensus.cc:2238] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.873085 1167 raft_consensus.cc:2267] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.873950 1167 ts_tablet_manager.cc:1905] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:55.867015 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9aeca9975f1945ad956f47db1ea16a73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9aeca9975f1945ad956f47db1ea16a73 already in progress: creating tablet
I20250115 14:51:55.866904 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_21fa1a9206d64c8bb19a405f37518a65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.867704 447 master_service.cc:946] ReplaceTablet: received request to replace tablet bf909c84eb314d95b73b6de82aebe9f2 from {username='slave'} at 127.0.0.1:47464
W20250115 14:51:55.874895 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21fa1a9206d64c8bb19a405f37518a65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:55.875907 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d6a9fb932c5b4545a4609d568f0e5494 because the tablet was not found. No further retry: Not found: Tablet not found: d6a9fb932c5b4545a4609d568f0e5494
I20250115 14:51:55.876123 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d243daba84ff4befa0c4e7bac5fe7476 (table default.Table_c06f4c91e37e4b6f95521174e05efb24 [id=9a6ed225ff3143f5bd2fc0f25324d322]) successfully deleted
I20250115 14:51:55.880260 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 3ae60842a2d4486eb7651c3f710ee1e8 (table default.Table_edeb6add07054dcba5df71224ae8675c [id=ad886fc755c2448c93964215e2fd5903]) successfully deleted
I20250115 14:51:55.880537 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 745153c106844076bb020f9ed026c97a (table default.Table_031dcfe40eda4ff49338ab79a0cf9844 [id=9cda5b5b1bcf47dfbc5b5a8dcc69b44a]) successfully deleted
W20250115 14:51:55.880772 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b92231509c754e8dbfc564813a76a123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b92231509c754e8dbfc564813a76a123 already in progress: creating tablet
I20250115 14:51:55.882798 1167 ts_tablet_manager.cc:1918] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:55.882934 1167 log.cc:1198] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/fad6f9f7f853418e85353edfd38cc80e
I20250115 14:51:55.883356 1167 ts_tablet_manager.cc:1939] T fad6f9f7f853418e85353edfd38cc80e P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
W20250115 14:51:55.883693 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3a6b57353484cdf8f2dfde2738ae0c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:55.899986 983 consensus_peers.cc:487] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 -> Peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Couldn't send request to peer 73c86158cc2641f182ea50a06634592d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58cca6072e4e4f608c6a45be279304f8. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:55.903297 1209 raft_consensus.cc:491] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.903496 1209 raft_consensus.cc:513] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.904165 1209 leader_election.cc:290] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:51:55.906855 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75788f8eb36b446e927197f624fbfdf5 consensus state has no leader
I20250115 14:51:55.908856 1165 ts_tablet_manager.cc:1918] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250115 14:51:55.909498 984 consensus_peers.cc:487] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:55.910676 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685a04e7454f6592f6ab83c356f466" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:55.910905 1168 tablet_bootstrap.cc:492] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.911027 1162 tablet_bootstrap.cc:492] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.911070 1168 ts_tablet_manager.cc:1397] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.019s
W20250115 14:51:55.911201 847 leader_election.cc:343] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.911798 1165 log.cc:1198] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/31d95c9359e54f1a88a07d5c5cce7fa1
I20250115 14:51:55.911834 1168 raft_consensus.cc:357] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.912053 1168 raft_consensus.cc:383] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.912120 1168 raft_consensus.cc:738] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.912433 1168 consensus_queue.cc:260] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.912938 1168 ts_tablet_manager.cc:1428] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:55.913295 1168 tablet_bootstrap.cc:492] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.914871 1168 tablet_bootstrap.cc:654] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.916709 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_76ab87d8a84c47008813112669e3800b" } modify_external_catalogs: true
I20250115 14:51:55.916947 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_76ab87d8a84c47008813112669e3800b" } modify_external_catalogs: true
I20250115 14:51:55.918251 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d1345e860a054b708e11f2f64347a3d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.918459 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet a188e8d22b794055b68ca3bfbe20e927
I20250115 14:51:55.918823 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b0a8c96252014e9298515a1f5ff14410"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.919281 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0a8c96252014e9298515a1f5ff14410 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.911823 1162 ts_tablet_manager.cc:1397] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.005s
W20250115 14:51:55.918799 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1345e860a054b708e11f2f64347a3d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.920284 1162 raft_consensus.cc:357] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.920938 1162 raft_consensus.cc:383] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.921032 1162 raft_consensus.cc:738] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.921252 1162 consensus_queue.cc:260] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.921411 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet bf909c84eb314d95b73b6de82aebe9f2
I20250115 14:51:55.921765 1162 ts_tablet_manager.cc:1428] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250115 14:51:55.921851 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b60ff80679a470cb4bdced71cc14195 consensus state has no leader
I20250115 14:51:55.922022 447 catalog_manager.cc:6596] ReplaceTablet: tablet bf909c84eb314d95b73b6de82aebe9f2 deleted and replaced by tablet 3a0d97fa57484b9d8cb879fedda61612
I20250115 14:51:55.922071 1162 tablet_bootstrap.cc:492] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.922226 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_21621e400c244baab7e990067052f131"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:55.922334 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b5c4f807878b4872bb3a73a6f9532957" } new_table_name: "default.Table_0c9709a31af8402eb6539db333df51a0" modify_external_catalogs: true
W20250115 14:51:55.922825 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21621e400c244baab7e990067052f131 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.924233 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0291acf45d75421b8b1c6311560e7f0c
I20250115 14:51:55.924631 446 catalog_manager.cc:6596] ReplaceTablet: tablet 0291acf45d75421b8b1c6311560e7f0c deleted and replaced by tablet f90e4edc7aea43268d581b441b788f27
I20250115 14:51:55.924845 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 5703fc5995184298bf72f9e43efb41f5 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.926362 1168 tablet_bootstrap.cc:492] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:55.926510 1168 ts_tablet_manager.cc:1397] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20250115 14:51:55.926926 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_854a113be33b4b698a50ae5bd10a2fff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:55.927352 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_854a113be33b4b698a50ae5bd10a2fff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:55.927328 1168 raft_consensus.cc:357] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.927531 1168 raft_consensus.cc:383] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.927626 1168 raft_consensus.cc:738] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.927851 1168 consensus_queue.cc:260] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:55.872373 1164 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.049s sys 0.046s
W20250115 14:51:55.928400 1164 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.558s user 0.049s sys 0.046s
I20250115 14:51:55.928418 1168 ts_tablet_manager.cc:1428] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250115 14:51:55.928594 1166 log.cc:1198] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/6326b1526191464485855b657e34a2b9
I20250115 14:51:55.928985 1166 ts_tablet_manager.cc:1939] T 6326b1526191464485855b657e34a2b9 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:55.930073 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fbd024b67b794f99b66433ffbb89184c" } modify_external_catalogs: true
I20250115 14:51:55.930238 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fbd024b67b794f99b66433ffbb89184c" } modify_external_catalogs: true
I20250115 14:51:55.882862 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de6f6f014074c3d8775cb5e4121ccb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:55.897804 754 tablet_service.cc:1467] Processing CreateTablet for tablet fbf94077e0e343b8938b081102dfd2af (DEFAULT_TABLE table=default.Table_c3b50255eb3e4642abaa02355b819e25 [id=00a89ba1b93f4132aae8fe6e748df77e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.931857 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbf94077e0e343b8938b081102dfd2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.933588 1166 tablet_replica.cc:331] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:55.933897 1166 raft_consensus.cc:2238] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.925915 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685a04e7454f6592f6ab83c356f466" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:55.934466 848 leader_election.cc:343] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:55.934603 848 leader_election.cc:304] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:55.935012 1209 raft_consensus.cc:2743] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:55.936406 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de6f6f014074c3d8775cb5e4121ccb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.938529 1165 ts_tablet_manager.cc:1939] T 31d95c9359e54f1a88a07d5c5cce7fa1 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:55.938638 1162 tablet_bootstrap.cc:654] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.938966 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 210) took 1475 ms. Trace:
I20250115 14:51:55.939285 1165 rpcz_store.cc:276] 0115 14:51:54.463475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:54.463740 (+ 265us) service_pool.cc:225] Handling call
0115 14:51:55.236838 (+773098us) ts_tablet_manager.cc:1145] Deleting tablet 31d95c9359e54f1a88a07d5c5cce7fa1
0115 14:51:55.236847 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:55.527374 (+290527us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:55.863958 (+336584us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:55.938923 (+ 74965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":701592}
I20250115 14:51:55.939762 1165 tablet_replica.cc:331] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:55.939743 878 tablet_service.cc:1467] Processing CreateTablet for tablet 37ad2b7350294375bd30fb4b46906491 (DEFAULT_TABLE table=default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.939996 1165 raft_consensus.cc:2238] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.940207 1165 raft_consensus.cc:2267] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.941195 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 37ad2b7350294375bd30fb4b46906491 (DEFAULT_TABLE table=default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.941618 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37ad2b7350294375bd30fb4b46906491. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.946816 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 3e990a2920b542f69a8bdd022db6e7e2 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:55.947913 1165 ts_tablet_manager.cc:1905] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.948310 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37ad2b7350294375bd30fb4b46906491. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.948453 1168 tablet_bootstrap.cc:492] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:55.948801 1549 raft_consensus.cc:491] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:55.948942 1549 raft_consensus.cc:513] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.949599 1549 leader_election.cc:290] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.950292 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4821224ac8f1472d8b1e8afc06a118a3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:55.950493 913 raft_consensus.cc:2463] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:55.950881 983 leader_election.cc:304] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:55.951141 1549 raft_consensus.cc:2798] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:55.951217 1549 raft_consensus.cc:491] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:55.951303 1549 raft_consensus.cc:3054] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.948570 1162 tablet_bootstrap.cc:492] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:55.952427 1162 ts_tablet_manager.cc:1397] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250115 14:51:55.952917 1168 tablet_bootstrap.cc:654] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.953178 1162 raft_consensus.cc:357] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.953353 1162 raft_consensus.cc:383] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.953429 1162 raft_consensus.cc:738] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.953603 1162 consensus_queue.cc:260] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.954010 1162 ts_tablet_manager.cc:1428] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:51:55.954211 1164 tablet_bootstrap.cc:492] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:55.954273 1162 tablet_bootstrap.cc:492] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:55.954402 1164 ts_tablet_manager.cc:1397] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.587s user 0.051s sys 0.046s
I20250115 14:51:55.956494 1162 tablet_bootstrap.cc:654] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:55.961874 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de6f6f014074c3d8775cb5e4121ccb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.963477 1165 ts_tablet_manager.cc:1918] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:51:55.964746 1549 raft_consensus.cc:513] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:55.965718 1165 log.cc:1198] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d6a9fb932c5b4545a4609d568f0e5494
I20250115 14:51:55.966133 1165 ts_tablet_manager.cc:1939] T d6a9fb932c5b4545a4609d568f0e5494 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:55.966931 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 211) took 1231 ms. Trace:
I20250115 14:51:55.967309 1549 leader_election.cc:290] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:55.968569 1578 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250115 14:51:55.967067 1165 rpcz_store.cc:276] 0115 14:51:54.735803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:54.735858 (+ 55us) service_pool.cc:225] Handling call
0115 14:51:55.939691 (+1203833us) ts_tablet_manager.cc:1145] Deleting tablet d6a9fb932c5b4545a4609d568f0e5494
0115 14:51:55.939701 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:55.951569 (+ 11868us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:55.963355 (+ 11786us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:55.966848 (+ 3493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1203627}
I20250115 14:51:55.969419 1165 tablet_replica.cc:331] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:51:55.969425 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3e990a2920b542f69a8bdd022db6e7e2 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3e990a2920b542f69a8bdd022db6e7e2 consensus state has no leader
I20250115 14:51:55.970117 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 31d95c9359e54f1a88a07d5c5cce7fa1 (table default.Table_9984f5b933e4431ba6201377446f3026 [id=db3a3335dd864c7384b2a414c31a6d2b]) successfully deleted
I20250115 14:51:55.970244 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 0291acf45d75421b8b1c6311560e7f0c with delete_type TABLET_DATA_DELETED (replaced by tablet f90e4edc7aea43268d581b441b788f27) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.970307 1165 raft_consensus.cc:2238] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:55.970357 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d6a9fb932c5b4545a4609d568f0e5494 (table default.Table_8f6e7d6b890e4a449f170811c030b6f5 [id=3f3fecceceb3464bb12462bb3041b458]) successfully deleted
I20250115 14:51:55.970533 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet fad6f9f7f853418e85353edfd38cc80e (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]) successfully deleted
I20250115 14:51:55.970623 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet bf909c84eb314d95b73b6de82aebe9f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0d97fa57484b9d8cb879fedda61612) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:55.970777 1165 raft_consensus.cc:2267] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.970825 1164 raft_consensus.cc:357] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.971073 1164 raft_consensus.cc:383] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:55.971305 1165 ts_tablet_manager.cc:1905] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.971406 1164 raft_consensus.cc:738] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:55.971652 1164 consensus_queue.cc:260] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:55.968971 1023 tablet_service.cc:1467] Processing CreateTablet for tablet fbf94077e0e343b8938b081102dfd2af (DEFAULT_TABLE table=default.Table_c3b50255eb3e4642abaa02355b819e25 [id=00a89ba1b93f4132aae8fe6e748df77e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.973089 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbf94077e0e343b8938b081102dfd2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.976732 1408 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" instance_seqno: 1736952710571245) as {username='slave'} at 127.0.104.3:39235; Asking this server to re-register.
I20250115 14:51:55.979116 1095 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:55.980207 1164 ts_tablet_manager.cc:1428] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250115 14:51:55.981582 1164 tablet_bootstrap.cc:492] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:55.981947 1165 ts_tablet_manager.cc:1918] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:55.982005 1408 ts_manager.cc:194] Registered new tserver with Master: c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:55.982059 1165 log.cc:1198] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f312f1e3a33a404ca7284c713859f156
I20250115 14:51:55.982383 1165 ts_tablet_manager.cc:1939] T f312f1e3a33a404ca7284c713859f156 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:55.983084 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 215) took 1162 ms. Trace:
I20250115 14:51:55.983194 1165 rpcz_store.cc:276] 0115 14:51:54.820969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:54.821145 (+ 176us) service_pool.cc:225] Handling call
0115 14:51:55.969328 (+1148183us) ts_tablet_manager.cc:1145] Deleting tablet f312f1e3a33a404ca7284c713859f156
0115 14:51:55.969338 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:55.974849 (+ 5511us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:55.981828 (+ 6979us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:55.983044 (+ 1216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147989}
I20250115 14:51:55.985862 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4821224ac8f1472d8b1e8afc06a118a3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:55.986018 1165 tablet_replica.cc:331] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:55.986096 913 raft_consensus.cc:3054] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:55.986208 1165 raft_consensus.cc:2238] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:55.986240 881 tablet_service.cc:1467] Processing CreateTablet for tablet fbf94077e0e343b8938b081102dfd2af (DEFAULT_TABLE table=default.Table_c3b50255eb3e4642abaa02355b819e25 [id=00a89ba1b93f4132aae8fe6e748df77e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:55.986475 1165 pending_rounds.cc:62] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:55.986482 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 0291acf45d75421b8b1c6311560e7f0c with delete_type TABLET_DATA_DELETED (replaced by tablet f90e4edc7aea43268d581b441b788f27) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.986560 1165 pending_rounds.cc:69] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558319316033536 op_type: NO_OP noop_request { }
I20250115 14:51:55.986677 1165 raft_consensus.cc:2883] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:55.986750 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbf94077e0e343b8938b081102dfd2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:55.986781 1165 raft_consensus.cc:2267] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.987303 1165 ts_tablet_manager.cc:1905] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.987524 888 tablet_service.cc:1514] Processing DeleteTablet for tablet bf909c84eb314d95b73b6de82aebe9f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0d97fa57484b9d8cb879fedda61612) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:55.988879 1167 tablet_replica.cc:331] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:55.989593 1167 raft_consensus.cc:2238] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:55.989941 1167 raft_consensus.cc:2267] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:55.990797 1167 ts_tablet_manager.cc:1905] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:55.991984 1561 tablet.cc:1721] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:55.993105 1517 tablet.cc:1721] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:55.997538 913 raft_consensus.cc:2463] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:55.997973 983 leader_election.cc:304] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:55.998016 1167 ts_tablet_manager.cc:1918] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:55.998157 1167 log.cc:1198] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/bf909c84eb314d95b73b6de82aebe9f2
I20250115 14:51:55.998296 1231 raft_consensus.cc:2798] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:55.998430 1231 raft_consensus.cc:695] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:55.998556 1167 ts_tablet_manager.cc:1939] T bf909c84eb314d95b73b6de82aebe9f2 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:55.998726 1231 consensus_queue.cc:237] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.006368 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4821224ac8f1472d8b1e8afc06a118a3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:56.007041 984 leader_election.cc:343] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.009902 1165 ts_tablet_manager.cc:1918] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:56.010046 1165 log.cc:1198] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/58cca6072e4e4f608c6a45be279304f8
I20250115 14:51:56.010440 1165 ts_tablet_manager.cc:1939] T 58cca6072e4e4f608c6a45be279304f8 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.011101 1164 tablet_bootstrap.cc:654] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.011214 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 216) took 1190 ms. Trace:
I20250115 14:51:56.011329 1165 rpcz_store.cc:276] 0115 14:51:54.821099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:54.821371 (+ 272us) service_pool.cc:225] Handling call
0115 14:51:55.985948 (+1164577us) ts_tablet_manager.cc:1145] Deleting tablet 58cca6072e4e4f608c6a45be279304f8
0115 14:51:55.985958 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:56.006568 (+ 20610us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.009754 (+ 3186us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.011179 (+ 1425us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164438}
I20250115 14:51:56.012380 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4821224ac8f1472d8b1e8afc06a118a3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:56.013136 984 leader_election.cc:343] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.021900 1165 tablet_replica.cc:331] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.022197 1165 raft_consensus.cc:2238] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.023283 1165 raft_consensus.cc:2267] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.023893 1165 ts_tablet_manager.cc:1905] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.030041 1408 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "73c86158cc2641f182ea50a06634592d" instance_seqno: 1736952710099038) as {username='slave'} at 127.0.104.2:34861; Asking this server to re-register.
W20250115 14:51:55.970758 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e932d5b820f94130967eccce9e1542fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e932d5b820f94130967eccce9e1542fa already in progress: creating tablet
W20250115 14:51:56.031955 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 61219.2us
W20250115 14:51:56.032368 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ac57fa232ae4462793b021b60ede0973 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac57fa232ae4462793b021b60ede0973 already in progress: creating tablet
W20250115 14:51:56.032670 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e50462626f914fb1b0f440c48f30e72d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e50462626f914fb1b0f440c48f30e72d already in progress: creating tablet
I20250115 14:51:56.035115 1408 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" instance_seqno: 1736952709640711) as {username='slave'} at 127.0.104.1:55115; Asking this server to re-register.
I20250115 14:51:56.036068 822 heartbeater.cc:463] Registering TS with master...
I20250115 14:51:56.037003 1165 ts_tablet_manager.cc:1918] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.038589 1408 ts_manager.cc:194] Registered new tserver with Master: d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127)
I20250115 14:51:56.038853 1165 log.cc:1198] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d1fa787e364d4ca6a4b756a8f4beba3b
I20250115 14:51:56.039255 1165 ts_tablet_manager.cc:1939] T d1fa787e364d4ca6a4b756a8f4beba3b P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.039868 1166 raft_consensus.cc:2267] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.040189 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 225) took 1183 ms. Trace:
I20250115 14:51:56.040652 1166 ts_tablet_manager.cc:1905] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.040830 1165 rpcz_store.cc:276] 0115 14:51:54.856959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:54.857573 (+ 614us) service_pool.cc:225] Handling call
0115 14:51:56.021804 (+1164231us) ts_tablet_manager.cc:1145] Deleting tablet d1fa787e364d4ca6a4b756a8f4beba3b
0115 14:51:56.021816 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:56.033770 (+ 11954us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.036892 (+ 3122us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.040129 (+ 3237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1164009}
W20250115 14:51:56.048738 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 79aad920dd9744d9b87a8234c92232d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79aad920dd9744d9b87a8234c92232d8 already in progress: creating tablet
W20250115 14:51:55.990206 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:40227, user_credentials={real_user=slave}} blocked reactor thread for 77594.2us
I20250115 14:51:56.050180 1165 tablet_replica.cc:331] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.050470 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 0291acf45d75421b8b1c6311560e7f0c with delete_type TABLET_DATA_DELETED (replaced by tablet f90e4edc7aea43268d581b441b788f27) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.050725 743 tablet_service.cc:1514] Processing DeleteTablet for tablet f224f857b1644596853acd9e3a2a4a5f with delete_type TABLET_DATA_DELETED (replaced by tablet e50462626f914fb1b0f440c48f30e72d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.052448 1165 raft_consensus.cc:2238] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.052737 1165 raft_consensus.cc:2267] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.053349 1165 ts_tablet_manager.cc:1905] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.056347 754 tablet_service.cc:1514] Processing DeleteTablet for tablet bf909c84eb314d95b73b6de82aebe9f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a0d97fa57484b9d8cb879fedda61612) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.057243 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 6326b1526191464485855b657e34a2b9 (table default.Table_662e371503c24e6cacf91cb2afe19d53 [id=ade9599dbabb4c298debe2172859adcc]) successfully deleted
I20250115 14:51:56.031579 959 heartbeater.cc:463] Registering TS with master...
W20250115 14:51:56.059461 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet bc8337d181d44ac7bb82551a677b6831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8337d181d44ac7bb82551a677b6831 already in progress: creating tablet
I20250115 14:51:56.059963 1165 ts_tablet_manager.cc:1918] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:56.060478 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0de6f6f014074c3d8775cb5e4121ccb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0de6f6f014074c3d8775cb5e4121ccb0 already in progress: creating tablet
I20250115 14:51:56.061456 1165 log.cc:1198] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b027233cfd314d0ebda472b4588d3f00
I20250115 14:51:56.062429 1165 ts_tablet_manager.cc:1939] T b027233cfd314d0ebda472b4588d3f00 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.064277 1168 tablet_bootstrap.cc:492] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.065254 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 226) took 1208 ms. Trace:
I20250115 14:51:56.065421 1165 rpcz_store.cc:276] 0115 14:51:54.857059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:54.857826 (+ 767us) service_pool.cc:225] Handling call
0115 14:51:56.049833 (+1192007us) ts_tablet_manager.cc:1145] Deleting tablet b027233cfd314d0ebda472b4588d3f00
0115 14:51:56.049848 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:56.056748 (+ 6900us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.059844 (+ 3096us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.065211 (+ 5367us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191211}
I20250115 14:51:56.066125 1165 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 232) took 1130 ms. Trace:
I20250115 14:51:56.066298 1165 rpcz_store.cc:276] 0115 14:51:54.935747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:54.935938 (+ 191us) service_pool.cc:225] Handling call
0115 14:51:56.065976 (+1130038us) ts_tablet_manager.cc:1145] Deleting tablet 276abefcab6a422aa6467f00864c01f4
0115 14:51:56.065986 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:56.066085 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1129746}
I20250115 14:51:56.067190 1168 ts_tablet_manager.cc:1397] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.012s
I20250115 14:51:56.068006 1168 raft_consensus.cc:357] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.068250 1168 raft_consensus.cc:383] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.068370 1168 raft_consensus.cc:738] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
W20250115 14:51:56.069033 983 consensus_peers.cc:487] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 -> Peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Couldn't send request to peer 73c86158cc2641f182ea50a06634592d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe0c1c829aff438a94bc41609bdb7841. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:56.069370 1168 consensus_queue.cc:260] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.071184 1168 ts_tablet_manager.cc:1428] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:51:56.071817 1168 tablet_bootstrap.cc:492] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:56.069036 1165 tablet_replica.cc:331] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.072598 1165 raft_consensus.cc:2238] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.072827 1165 raft_consensus.cc:2267] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.073405 1165 ts_tablet_manager.cc:1905] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.074492 1168 tablet_bootstrap.cc:654] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:56.073913 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0291acf45d75421b8b1c6311560e7f0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0291acf45d75421b8b1c6311560e7f0c already in progress: creating tablet
W20250115 14:51:56.076155 440 catalog_manager.cc:4951] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): alter failed for tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]): Illegal state: Tablet not RUNNING: STOPPING
I20250115 14:51:56.078497 1408 ts_manager.cc:194] Registered new tserver with Master: 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.085510 1165 ts_tablet_manager.cc:1918] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.088971 1164 tablet_bootstrap.cc:492] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:56.089146 1164 ts_tablet_manager.cc:1397] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.003s
I20250115 14:51:56.089180 1165 log.cc:1198] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f55c75e5febb407398b0bc06a1db9b1a
I20250115 14:51:56.089905 1165 ts_tablet_manager.cc:1939] T f55c75e5febb407398b0bc06a1db9b1a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.090251 1162 tablet_bootstrap.cc:492] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.090293 1164 raft_consensus.cc:357] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.090415 1162 ts_tablet_manager.cc:1397] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.136s user 0.024s sys 0.061s
I20250115 14:51:56.090504 1164 raft_consensus.cc:383] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.090586 1164 raft_consensus.cc:738] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.090857 1164 consensus_queue.cc:260] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.091715 1162 raft_consensus.cc:357] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.091934 1162 raft_consensus.cc:383] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.092025 1162 raft_consensus.cc:738] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.092244 1162 consensus_queue.cc:260] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.092495 1165 tablet_replica.cc:331] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.092772 1162 ts_tablet_manager.cc:1428] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250115 14:51:56.092957 1165 raft_consensus.cc:2238] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:56.093011 1168 tablet_bootstrap.cc:492] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.093142 1168 ts_tablet_manager.cc:1397] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250115 14:51:56.093384 1165 pending_rounds.cc:62] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:56.093492 1165 pending_rounds.cc:69] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558320846712832 op_type: NO_OP noop_request { }
I20250115 14:51:56.093678 1165 raft_consensus.cc:2883] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:56.093840 1165 raft_consensus.cc:2267] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.093863 1168 raft_consensus.cc:357] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.094058 1168 raft_consensus.cc:383] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.094146 1168 raft_consensus.cc:738] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.094396 1168 consensus_queue.cc:260] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.095268 1162 tablet_bootstrap.cc:492] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.095973 1168 ts_tablet_manager.cc:1428] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:56.095293 1165 ts_tablet_manager.cc:1905] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.096876 1162 tablet_bootstrap.cc:654] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.102026 1164 ts_tablet_manager.cc:1428] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250115 14:51:56.104988 1165 ts_tablet_manager.cc:1918] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:56.105173 1165 log.cc:1198] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/fe0c1c829aff438a94bc41609bdb7841
I20250115 14:51:56.105685 1165 ts_tablet_manager.cc:1939] T fe0c1c829aff438a94bc41609bdb7841 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.106724 1165 tablet_replica.cc:331] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.106972 1165 raft_consensus.cc:2238] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:56.107314 1165 pending_rounds.cc:62] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:56.107416 1165 pending_rounds.cc:69] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558319651147776 op_type: NO_OP noop_request { }
I20250115 14:51:56.107578 1165 raft_consensus.cc:2883] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:56.107702 1165 raft_consensus.cc:2267] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.108304 1165 ts_tablet_manager.cc:1905] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.110960 1164 tablet_bootstrap.cc:492] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:56.112752 1164 tablet_bootstrap.cc:654] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:56.113562 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d86075ae5b894d5986ac4bd538f4d42a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d86075ae5b894d5986ac4bd538f4d42a already in progress: creating tablet
I20250115 14:51:56.114060 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]) successfully deleted
W20250115 14:51:56.114827 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 52721.1us
I20250115 14:51:56.115008 1165 ts_tablet_manager.cc:1918] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:56.115197 1165 log.cc:1198] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3fd72a0d586d4a2c817f6a39daa07063
I20250115 14:51:56.115649 1165 ts_tablet_manager.cc:1939] T 3fd72a0d586d4a2c817f6a39daa07063 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.117291 1165 tablet_replica.cc:331] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.117525 1165 raft_consensus.cc:2238] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:56.117836 1165 pending_rounds.cc:62] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:56.117954 1165 pending_rounds.cc:69] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558322747199488 op_type: NO_OP noop_request { }
I20250115 14:51:56.118139 1165 raft_consensus.cc:2883] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:56.118304 1165 raft_consensus.cc:2267] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.119194 1165 ts_tablet_manager.cc:1905] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.121675 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet cea16f889c73400a85b9553c1975457e
I20250115 14:51:56.122287 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_10deaf92a667422fa029a731f755dfef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.122884 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10deaf92a667422fa029a731f755dfef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.124441 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4ec686b46a094818bd820c4f3854220b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.124938 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ec686b46a094818bd820c4f3854220b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:56.126087 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d71a1ecd900942e5ae788c9a28c70a7a has no consensus state
I20250115 14:51:56.126261 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3e990a2920b542f69a8bdd022db6e7e2
I20250115 14:51:56.126427 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_27cd4d793c8f4e77aa531c06dfa74a2d" } modify_external_catalogs: true
I20250115 14:51:56.126612 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_27cd4d793c8f4e77aa531c06dfa74a2d" } modify_external_catalogs: true
I20250115 14:51:56.126672 455 catalog_manager.cc:6596] ReplaceTablet: tablet 3e990a2920b542f69a8bdd022db6e7e2 deleted and replaced by tablet 4ee8b63cc21145d8993b058d8c6d23c7
I20250115 14:51:56.126933 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7171658b8daf4ee3868f00408ce43d84" } modify_external_catalogs: true
I20250115 14:51:56.127091 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7171658b8daf4ee3868f00408ce43d84" } modify_external_catalogs: true
I20250115 14:51:56.127178 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4b90205ef15f470a99c013e7f2cf99c6 (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]). Expected version 3 got 0
I20250115 14:51:56.127417 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 86f0144904ae4008945ccf325039dac1 (table default.Table_39eb08eb474049f2a2fbe741ef730d68 [id=ab6f49066e884652b77a0b9a791575c7]). Expected version 1 got 0
I20250115 14:51:56.128163 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet bb792ca1b3f54def92f504ec8682d2c7
I20250115 14:51:56.128798 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_50d67783a2d6443097b0027856df1225" } new_table_name: "default.Table_44a23f5fdfb446a59f551a2a7a4ca2ce" modify_external_catalogs: true
I20250115 14:51:56.130529 453 master_service.cc:946] ReplaceTablet: received request to replace tablet e3fbde011216426cb41c51fe359dcb5f from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:56.131063 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1653219949dc4fdeb257d0ba8c88c231" } new_table_name: "default.Table_d810e40dae3d42e38f38ebf32e62305e" modify_external_catalogs: true
I20250115 14:51:56.132187 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5703fc5995184298bf72f9e43efb41f5
I20250115 14:51:56.132639 446 catalog_manager.cc:6596] ReplaceTablet: tablet 5703fc5995184298bf72f9e43efb41f5 deleted and replaced by tablet 8e6cc08065494d9e85eeff5044b44ba6
I20250115 14:51:56.135738 1162 tablet_bootstrap.cc:492] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.135897 1162 ts_tablet_manager.cc:1397] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.041s user 0.019s sys 0.006s
I20250115 14:51:56.136770 1162 raft_consensus.cc:357] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.136981 1162 raft_consensus.cc:383] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.137087 1162 raft_consensus.cc:738] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.137323 1162 consensus_queue.cc:260] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.138198 1168 tablet_bootstrap.cc:492] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:56.139183 1165 ts_tablet_manager.cc:1918] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:56.139461 1165 log.cc:1198] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3ae60842a2d4486eb7651c3f710ee1e8
I20250115 14:51:56.139912 1162 ts_tablet_manager.cc:1428] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250115 14:51:56.140044 1165 ts_tablet_manager.cc:1939] T 3ae60842a2d4486eb7651c3f710ee1e8 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.140200 1162 tablet_bootstrap.cc:492] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.140472 1168 tablet_bootstrap.cc:654] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.141522 1165 tablet_replica.cc:331] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.141834 1165 raft_consensus.cc:2238] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.142059 1165 raft_consensus.cc:2267] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.142642 1165 ts_tablet_manager.cc:1905] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.141839 1162 tablet_bootstrap.cc:654] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.145634 1168 tablet_bootstrap.cc:492] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.145802 1168 ts_tablet_manager.cc:1397] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250115 14:51:56.146534 1168 raft_consensus.cc:357] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.146759 1168 raft_consensus.cc:383] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.146854 1168 raft_consensus.cc:738] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.147218 1168 consensus_queue.cc:260] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.147815 1168 ts_tablet_manager.cc:1428] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:56.148104 1168 tablet_bootstrap.cc:492] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:56.149569 1165 ts_tablet_manager.cc:1918] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.149834 1165 log.cc:1198] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a6b76cb343fb4456ac07fd3c6e9888de
I20250115 14:51:56.150192 1168 tablet_bootstrap.cc:654] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.150209 1165 ts_tablet_manager.cc:1939] T a6b76cb343fb4456ac07fd3c6e9888de P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.151289 1165 tablet_replica.cc:331] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.151526 1165 raft_consensus.cc:2238] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.151758 1165 raft_consensus.cc:2267] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.152417 1165 ts_tablet_manager.cc:1905] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.153638 1162 tablet_bootstrap.cc:492] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.153791 1162 ts_tablet_manager.cc:1397] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.001s
I20250115 14:51:56.154531 1162 raft_consensus.cc:357] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.154762 1162 raft_consensus.cc:383] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.154830 1162 raft_consensus.cc:738] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.155061 1162 consensus_queue.cc:260] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.155965 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f312f1e3a33a404ca7284c713859f156 (table default.Table_2752e5f3a4ca4c41a614738b9354d962 [id=1c779c8a53f54707abe1d619b30a912e]) successfully deleted
I20250115 14:51:56.156178 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 58cca6072e4e4f608c6a45be279304f8 (table default.Table_ca27ab47d077491489383629eaaf1437 [id=5ec280e869eb4ba6bec54fbc22af1409]) successfully deleted
I20250115 14:51:56.158660 1162 ts_tablet_manager.cc:1428] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:56.158818 1165 ts_tablet_manager.cc:1918] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.159009 1165 log.cc:1198] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ea0f104c1a71404591072e7a36b276b0
I20250115 14:51:56.159461 1165 ts_tablet_manager.cc:1939] T ea0f104c1a71404591072e7a36b276b0 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.159982 1162 tablet_bootstrap.cc:492] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.161217 1165 tablet_replica.cc:331] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.161459 1165 raft_consensus.cc:2238] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:56.116590 1173 raft_consensus.cc:491] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.161753 1173 raft_consensus.cc:513] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.162441 1173 leader_election.cc:290] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.162597 1173 raft_consensus.cc:491] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.162724 1173 raft_consensus.cc:513] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.163208 1173 leader_election.cc:290] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.163614 1165 raft_consensus.cc:2267] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.164613 1165 ts_tablet_manager.cc:1905] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.164716 1162 tablet_bootstrap.cc:654] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.166712 1468 raft_consensus.cc:491] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.166878 1468 raft_consensus.cc:513] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.167482 1468 leader_election.cc:290] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.168196 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259ac764ca844df9b6c0ce1688a188da" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.168465 1047 raft_consensus.cc:2463] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:56.169081 847 leader_election.cc:304] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.169580 1468 raft_consensus.cc:2798] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.169672 1468 raft_consensus.cc:491] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.169760 1468 raft_consensus.cc:3054] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.171154 1165 ts_tablet_manager.cc:1918] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:56.171275 1165 log.cc:1198] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/fad6f9f7f853418e85353edfd38cc80e
I20250115 14:51:56.171666 1165 ts_tablet_manager.cc:1939] T fad6f9f7f853418e85353edfd38cc80e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.172544 1209 raft_consensus.cc:491] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.172701 1209 raft_consensus.cc:513] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.173323 1209 leader_election.cc:290] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.173919 1165 tablet_replica.cc:331] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.174038 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fb2d33f3c745beb4a4ba036e23a8e0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.174227 1165 raft_consensus.cc:2238] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:56.172245 1468 raft_consensus.cc:513] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.174377 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b027233cfd314d0ebda472b4588d3f00 (table default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]) successfully deleted
I20250115 14:51:56.174569 1165 pending_rounds.cc:62] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:56.174714 1165 pending_rounds.cc:69] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558323229859840 op_type: NO_OP noop_request { }
I20250115 14:51:56.174255 1047 raft_consensus.cc:2463] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:56.174943 1165 raft_consensus.cc:2883] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:56.175129 1165 raft_consensus.cc:2267] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.175216 847 leader_election.cc:304] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.175534 1209 raft_consensus.cc:2798] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.175629 1209 raft_consensus.cc:491] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.175729 1209 raft_consensus.cc:3054] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.176117 1165 ts_tablet_manager.cc:1905] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.178479 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259ac764ca844df9b6c0ce1688a188da" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:56.178674 1047 raft_consensus.cc:3054] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.181411 1047 raft_consensus.cc:2463] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:56.181771 847 leader_election.cc:304] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.182307 1165 ts_tablet_manager.cc:1918] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:56.182463 1165 log.cc:1198] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d243daba84ff4befa0c4e7bac5fe7476
I20250115 14:51:56.182945 1165 ts_tablet_manager.cc:1939] T d243daba84ff4befa0c4e7bac5fe7476 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.183369 1209 raft_consensus.cc:513] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.184018 1468 leader_election.cc:290] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.184186 1468 raft_consensus.cc:2798] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.184356 1468 raft_consensus.cc:695] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:56.184981 1165 tablet_replica.cc:331] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.185236 1165 raft_consensus.cc:2238] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.185483 1165 raft_consensus.cc:2267] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.185562 1209 leader_election.cc:290] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.184620 1468 consensus_queue.cc:237] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.186223 1165 ts_tablet_manager.cc:1905] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:56.189411 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 276abefcab6a422aa6467f00864c01f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 276abefcab6a422aa6467f00864c01f4 already in progress: creating tablet
W20250115 14:51:56.189729 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 5027fba8fed34db59b484918256abc45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5027fba8fed34db59b484918256abc45 already in progress: creating tablet
I20250115 14:51:56.192571 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259ac764ca844df9b6c0ce1688a188da" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.193171 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fb2d33f3c745beb4a4ba036e23a8e0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.193636 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "259ac764ca844df9b6c0ce1688a188da" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:56.193892 1165 ts_tablet_manager.cc:1918] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:56.194069 1165 log.cc:1198] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/bf909c84eb314d95b73b6de82aebe9f2
I20250115 14:51:56.194135 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fb2d33f3c745beb4a4ba036e23a8e0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:56.194555 1165 ts_tablet_manager.cc:1939] T bf909c84eb314d95b73b6de82aebe9f2 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.194849 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e990a2920b542f69a8bdd022db6e7e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4ee8b63cc21145d8993b058d8c6d23c7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.195883 754 tablet_service.cc:1514] Processing DeleteTablet for tablet bb792ca1b3f54def92f504ec8682d2c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.196341 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5703fc5995184298bf72f9e43efb41f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e6cc08065494d9e85eeff5044b44ba6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.196733 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 74bfda3861d148bc8b67ea071909d22d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:51 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.196853 1168 tablet_bootstrap.cc:492] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.197026 1168 ts_tablet_manager.cc:1397] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
W20250115 14:51:56.197188 848 leader_election.cc:343] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:56.197866 848 leader_election.cc:343] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.197835 1168 raft_consensus.cc:357] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.198024 1168 raft_consensus.cc:383] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.198108 1168 raft_consensus.cc:738] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.198309 1168 consensus_queue.cc:260] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:56.198436 848 leader_election.cc:343] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.198843 1168 ts_tablet_manager.cc:1428] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250115 14:51:56.198875 848 leader_election.cc:343] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.199118 1168 tablet_bootstrap.cc:492] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:51:56.199237 710 leader_election.cc:343] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 5d51e53b4ffa45bfbb9553d7a5870844
W20250115 14:51:56.199472 710 leader_election.cc:343] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 3a38a6bc32314396b497e7ae166a63f7
W20250115 14:51:56.199878 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet bd4102b1f83e414d93a6a68315e36d16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd4102b1f83e414d93a6a68315e36d16 already in progress: creating tablet
W20250115 14:51:56.200163 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet bc8337d181d44ac7bb82551a677b6831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc8337d181d44ac7bb82551a677b6831 already in progress: creating tablet
W20250115 14:51:56.200416 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e3fbde011216426cb41c51fe359dcb5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3fbde011216426cb41c51fe359dcb5f already in progress: creating tablet
I20250115 14:51:56.200755 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f55c75e5febb407398b0bc06a1db9b1a (table default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]) successfully deleted
I20250115 14:51:56.200795 1168 tablet_bootstrap.cc:654] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:56.201099 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d86075ae5b894d5986ac4bd538f4d42a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d86075ae5b894d5986ac4bd538f4d42a already in progress: creating tablet
I20250115 14:51:56.201313 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet fe0c1c829aff438a94bc41609bdb7841 (table default.Table_27cd4d793c8f4e77aa531c06dfa74a2d [id=a6994f260e1a4eada78d8c9e9fcf334e]) successfully deleted
I20250115 14:51:56.201576 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3fd72a0d586d4a2c817f6a39daa07063 (table default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]) successfully deleted
W20250115 14:51:56.201843 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9aeca9975f1945ad956f47db1ea16a73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9aeca9975f1945ad956f47db1ea16a73 already in progress: creating tablet
I20250115 14:51:56.202095 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3ae60842a2d4486eb7651c3f710ee1e8 (table default.Table_edeb6add07054dcba5df71224ae8675c [id=ad886fc755c2448c93964215e2fd5903]) successfully deleted
I20250115 14:51:56.202394 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a6b76cb343fb4456ac07fd3c6e9888de (table default.Table_91244a7b0fa14291bcc2e9a2d776b4ae [id=4a31d55ea43e4ce5bbda5b78fdbfffa6]) successfully deleted
I20250115 14:51:56.202625 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ea0f104c1a71404591072e7a36b276b0 (table default.Table_30d9bde332c24d2a94267ffd2605a7cf [id=717e74f1625e4a5a92c254f204df1629]) successfully deleted
I20250115 14:51:56.202845 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet fad6f9f7f853418e85353edfd38cc80e (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]) successfully deleted
W20250115 14:51:56.203081 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b92231509c754e8dbfc564813a76a123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b92231509c754e8dbfc564813a76a123 already in progress: creating tablet
W20250115 14:51:56.203383 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e50462626f914fb1b0f440c48f30e72d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e50462626f914fb1b0f440c48f30e72d already in progress: creating tablet
I20250115 14:51:56.204694 1231 raft_consensus.cc:491] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.204891 1231 raft_consensus.cc:513] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.205724 1231 leader_election.cc:290] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.210750 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1a38186c5042ff851f57979ab80dbc" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:56.211014 913 raft_consensus.cc:2463] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:56.211622 983 leader_election.cc:304] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.212167 1231 raft_consensus.cc:2798] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.212266 1231 raft_consensus.cc:491] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.212388 1231 raft_consensus.cc:3054] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.216670 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1a38186c5042ff851f57979ab80dbc" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:56.217355 984 leader_election.cc:343] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.216272 1231 raft_consensus.cc:513] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.219770 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1a38186c5042ff851f57979ab80dbc" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:56.220588 984 leader_election.cc:343] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.220871 1168 tablet_bootstrap.cc:492] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.221035 1168 ts_tablet_manager.cc:1397] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20250115 14:51:56.221540 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab1a38186c5042ff851f57979ab80dbc" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:56.221804 913 raft_consensus.cc:3054] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.221864 1168 raft_consensus.cc:357] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.222083 1168 raft_consensus.cc:383] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.222146 1168 raft_consensus.cc:738] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.222342 1168 consensus_queue.cc:260] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.222853 1168 ts_tablet_manager.cc:1428] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:56.223320 1168 tablet_bootstrap.cc:492] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:56.224982 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fb2d33f3c745beb4a4ba036e23a8e0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:56.225023 913 raft_consensus.cc:2463] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:56.225163 1168 tablet_bootstrap.cc:654] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.225170 1047 raft_consensus.cc:3054] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.225732 983 leader_election.cc:304] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.226073 1549 raft_consensus.cc:2798] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.226227 1549 raft_consensus.cc:695] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:56.226526 1549 consensus_queue.cc:237] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.228425 1047 raft_consensus.cc:2463] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:56.229844 847 leader_election.cc:304] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.231359 1468 raft_consensus.cc:2798] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.231554 1468 raft_consensus.cc:695] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:56.231760 1225 raft_consensus.cc:491] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.231905 1225 raft_consensus.cc:513] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.231890 1468 consensus_queue.cc:237] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.232511 1225 leader_election.cc:290] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.237519 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7580b6245b41628874d9dcbc3366e2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:56.237782 913 raft_consensus.cc:2463] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:56.238265 983 leader_election.cc:304] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.240324 1549 raft_consensus.cc:2798] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.240476 1549 raft_consensus.cc:491] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.240626 1549 raft_consensus.cc:3054] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.243613 1549 raft_consensus.cc:513] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.244886 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7580b6245b41628874d9dcbc3366e2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:56.245532 984 leader_election.cc:343] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.246428 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7580b6245b41628874d9dcbc3366e2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:56.246634 913 raft_consensus.cc:3054] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.249603 913 raft_consensus.cc:2463] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:56.250104 983 leader_election.cc:304] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.250438 1225 raft_consensus.cc:2798] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.250576 1225 raft_consensus.cc:695] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:56.250806 1225 consensus_queue.cc:237] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.255050 1168 tablet_bootstrap.cc:492] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.255208 1168 ts_tablet_manager.cc:1397] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.032s user 0.016s sys 0.012s
I20250115 14:51:56.256052 1168 raft_consensus.cc:357] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.256318 1168 raft_consensus.cc:383] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.256410 1168 raft_consensus.cc:738] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.256652 1168 consensus_queue.cc:260] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.257200 1168 ts_tablet_manager.cc:1428] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:56.257481 1168 tablet_bootstrap.cc:492] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:56.259212 1168 tablet_bootstrap.cc:654] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.266885 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7580b6245b41628874d9dcbc3366e2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:56.267741 984 leader_election.cc:343] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.270510 1168 tablet_bootstrap.cc:492] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.270732 1468 raft_consensus.cc:491] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.270905 1468 raft_consensus.cc:513] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.271883 1162 tablet_bootstrap.cc:492] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.272012 1468 leader_election.cc:290] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.272047 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4239747374aa4d6d866961430786677c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.272219 1168 ts_tablet_manager.cc:1397] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
W20250115 14:51:56.272745 847 leader_election.cc:343] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.273116 1168 raft_consensus.cc:357] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.273334 1168 raft_consensus.cc:383] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.273419 1168 raft_consensus.cc:738] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.273660 1168 consensus_queue.cc:260] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.272049 1162 ts_tablet_manager.cc:1397] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.112s user 0.049s sys 0.053s
I20250115 14:51:56.274176 1168 ts_tablet_manager.cc:1428] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:56.274468 1168 tablet_bootstrap.cc:492] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:56.276131 1168 tablet_bootstrap.cc:654] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.276666 1162 raft_consensus.cc:357] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.276868 1162 raft_consensus.cc:383] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.276959 1162 raft_consensus.cc:738] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.277182 1162 consensus_queue.cc:260] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.277721 1162 ts_tablet_manager.cc:1428] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20250115 14:51:56.279326 1166 ts_tablet_manager.cc:1918] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.279513 1166 log.cc:1198] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/13c00f03edc74af8b084bb236487a347
I20250115 14:51:56.279729 1225 raft_consensus.cc:491] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.279858 1225 raft_consensus.cc:513] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.280128 1166 ts_tablet_manager.cc:1939] T 13c00f03edc74af8b084bb236487a347 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:56.280485 1225 leader_election.cc:290] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.281020 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46776 (request call id 232) took 1041 ms. Trace:
I20250115 14:51:56.281208 1166 rpcz_store.cc:276] 0115 14:51:55.239322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:55.240572 (+ 1250us) service_pool.cc:225] Handling call
0115 14:51:55.933489 (+692917us) ts_tablet_manager.cc:1145] Deleting tablet 13c00f03edc74af8b084bb236487a347
0115 14:51:55.933496 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:56.069319 (+135823us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.279171 (+209852us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.280982 (+ 1811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":692664}
I20250115 14:51:56.282321 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcfab1e1f0349739eff636fb1b04806" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.282598 1162 tablet_bootstrap.cc:492] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:56.282817 984 leader_election.cc:343] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.283258 1166 tablet_replica.cc:331] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:56.283490 1166 raft_consensus.cc:2238] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.283442 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcfab1e1f0349739eff636fb1b04806" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:56.283658 913 raft_consensus.cc:2463] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:56.283717 1166 raft_consensus.cc:2267] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.284018 983 leader_election.cc:304] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.284251 1162 tablet_bootstrap.cc:654] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.284433 1225 raft_consensus.cc:2798] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.284452 1166 ts_tablet_manager.cc:1905] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.284559 1225 raft_consensus.cc:491] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.284662 1225 raft_consensus.cc:3054] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.287374 1225 raft_consensus.cc:513] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.289773 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcfab1e1f0349739eff636fb1b04806" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:56.290609 984 leader_election.cc:343] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.293933 1168 tablet_bootstrap.cc:492] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.294095 1225 leader_election.cc:290] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.294138 1168 ts_tablet_manager.cc:1397] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250115 14:51:56.294492 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcfab1e1f0349739eff636fb1b04806" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:56.294744 913 raft_consensus.cc:3054] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.294875 1168 raft_consensus.cc:357] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.295073 1168 raft_consensus.cc:383] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.295178 1168 raft_consensus.cc:738] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.295423 1168 consensus_queue.cc:260] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.297480 1168 ts_tablet_manager.cc:1428] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:56.297745 1168 tablet_bootstrap.cc:492] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:56.298105 913 raft_consensus.cc:2463] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:56.298547 983 leader_election.cc:304] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.299295 1168 tablet_bootstrap.cc:654] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.300072 1166 ts_tablet_manager.cc:1918] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.300227 1166 log.cc:1198] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/3a38a6bc32314396b497e7ae166a63f7
I20250115 14:51:56.300591 1166 ts_tablet_manager.cc:1939] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:56.301048 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46776 (request call id 233) took 1025 ms. Trace:
I20250115 14:51:56.301136 1166 rpcz_store.cc:276] 0115 14:51:55.275460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:55.291942 (+ 16482us) service_pool.cc:225] Handling call
0115 14:51:56.282344 (+990402us) ts_tablet_manager.cc:1145] Deleting tablet 3a38a6bc32314396b497e7ae166a63f7
0115 14:51:56.282353 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:56.287496 (+ 5143us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.292488 (+ 4992us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.300997 (+ 8509us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":990207}
I20250115 14:51:56.302004 1166 tablet_replica.cc:331] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:56.302182 1166 raft_consensus.cc:2238] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.302348 1166 raft_consensus.cc:2267] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.302840 1166 ts_tablet_manager.cc:1905] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.303550 1225 raft_consensus.cc:2798] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.304126 1225 raft_consensus.cc:695] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:56.304535 1225 consensus_queue.cc:237] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.305753 1549 leader_election.cc:290] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.306087 1231 leader_election.cc:290] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.310544 1162 tablet_bootstrap.cc:492] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.310709 1162 ts_tablet_manager.cc:1397] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.004s
I20250115 14:51:56.311370 1164 tablet_bootstrap.cc:492] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:56.311430 1162 raft_consensus.cc:357] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.311650 1162 raft_consensus.cc:383] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.311725 1162 raft_consensus.cc:738] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.311745 1164 ts_tablet_manager.cc:1397] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.201s user 0.002s sys 0.002s
I20250115 14:51:56.311961 1162 consensus_queue.cc:260] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.312482 1162 ts_tablet_manager.cc:1428] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:56.312757 1162 tablet_bootstrap.cc:492] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.314357 1162 tablet_bootstrap.cc:654] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.317073 1166 ts_tablet_manager.cc:1918] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.317193 1166 log.cc:1198] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/5d51e53b4ffa45bfbb9553d7a5870844
I20250115 14:51:56.317533 1166 ts_tablet_manager.cc:1939] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:56.317978 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46776 (request call id 235) took 1018 ms. Trace:
I20250115 14:51:56.318082 1166 rpcz_store.cc:276] 0115 14:51:55.299874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:55.323128 (+ 23254us) service_pool.cc:225] Handling call
0115 14:51:56.301383 (+978255us) ts_tablet_manager.cc:1145] Deleting tablet 5d51e53b4ffa45bfbb9553d7a5870844
0115 14:51:56.301394 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:56.305660 (+ 4266us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.316920 (+ 11260us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:56.317940 (+ 1020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":977768}
I20250115 14:51:56.318482 1166 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46776 (request call id 236) took 1013 ms. Trace:
I20250115 14:51:56.318588 1166 rpcz_store.cc:276] 0115 14:51:55.304908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:55.324944 (+ 20036us) service_pool.cc:225] Handling call
0115 14:51:56.318337 (+993393us) ts_tablet_manager.cc:1145] Deleting tablet 9aeca9975f1945ad956f47db1ea16a73
0115 14:51:56.318347 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:56.318452 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":993081}
I20250115 14:51:56.319895 1166 tablet_replica.cc:331] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:56.320093 1166 raft_consensus.cc:2238] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.320240 1166 raft_consensus.cc:2267] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.320899 1166 ts_tablet_manager.cc:1905] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.324100 1468 raft_consensus.cc:491] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.324280 1468 raft_consensus.cc:513] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.325089 1468 leader_election.cc:290] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.326030 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b90205ef15f470a99c013e7f2cf99c6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.326256 1047 raft_consensus.cc:2463] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:56.326886 847 leader_election.cc:304] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.327248 1468 raft_consensus.cc:2798] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.327358 1468 raft_consensus.cc:491] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.327436 1468 raft_consensus.cc:3054] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.328161 1162 tablet_bootstrap.cc:492] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.328311 1162 ts_tablet_manager.cc:1397] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20250115 14:51:56.329068 1162 raft_consensus.cc:357] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.329257 1162 raft_consensus.cc:383] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.329339 1162 raft_consensus.cc:738] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.329545 1162 consensus_queue.cc:260] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.330060 1162 ts_tablet_manager.cc:1428] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:56.330323 1162 tablet_bootstrap.cc:492] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.330946 1468 raft_consensus.cc:513] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.332356 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b90205ef15f470a99c013e7f2cf99c6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:56.332566 1047 raft_consensus.cc:3054] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.332719 1162 tablet_bootstrap.cc:654] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.335387 1047 raft_consensus.cc:2463] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:56.336124 847 leader_election.cc:304] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.336427 1594 raft_consensus.cc:2798] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.336591 1594 raft_consensus.cc:695] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:56.336788 1166 ts_tablet_manager.cc:1918] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.336938 1166 log.cc:1198] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f224f857b1644596853acd9e3a2a4a5f
I20250115 14:51:56.336897 1594 consensus_queue.cc:237] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.337268 1166 ts_tablet_manager.cc:1939] T f224f857b1644596853acd9e3a2a4a5f P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:56.337981 1468 leader_election.cc:290] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.338133 1166 tablet_replica.cc:331] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:56.338306 1166 raft_consensus.cc:2238] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.338513 1166 raft_consensus.cc:2267] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.340950 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4239747374aa4d6d866961430786677c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.341471 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b90205ef15f470a99c013e7f2cf99c6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:56.341928 710 leader_election.cc:343] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 5d51e53b4ffa45bfbb9553d7a5870844
I20250115 14:51:56.342049 710 leader_election.cc:304] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:56.342262 710 raft_consensus.cc:2681] T 5d51e53b4ffa45bfbb9553d7a5870844 P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250115 14:51:56.342947 710 leader_election.cc:343] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 3a38a6bc32314396b497e7ae166a63f7
I20250115 14:51:56.343091 710 leader_election.cc:304] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:56.343266 710 raft_consensus.cc:2681] T 3a38a6bc32314396b497e7ae166a63f7 P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250115 14:51:56.343962 848 leader_election.cc:343] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.344132 848 leader_election.cc:304] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.344573 1468 raft_consensus.cc:2743] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:56.344895 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b90205ef15f470a99c013e7f2cf99c6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:56.354408 1164 raft_consensus.cc:357] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.354637 1164 raft_consensus.cc:383] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.354732 1164 raft_consensus.cc:738] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.354966 1164 consensus_queue.cc:260] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.355495 1164 ts_tablet_manager.cc:1428] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.044s user 0.001s sys 0.001s
I20250115 14:51:56.356546 1468 raft_consensus.cc:491] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.356747 1468 raft_consensus.cc:513] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.357821 1468 leader_election.cc:290] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.358145 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1839edd0d49b467d8bb31b3001ff89eb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:56.358775 847 leader_election.cc:343] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.389285 1164 tablet_bootstrap.cc:492] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:56.391021 1164 tablet_bootstrap.cc:654] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:56.208571 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a73bd0141ccc4ec4b052b18b9fb62bb7 consensus state has no leader
W20250115 14:51:56.208868 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4239747374aa4d6d866961430786677c consensus state has no leader
I20250115 14:51:56.211303 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b59afef1daad4747a3cc11e5cd60c2c8
W20250115 14:51:56.339829 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:40227, user_credentials={real_user=slave}} blocked reactor thread for 150495us
W20250115 14:51:56.398734 848 leader_election.cc:343] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:56.399150 848 leader_election.cc:343] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.400694 1168 tablet_bootstrap.cc:492] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.400900 1168 ts_tablet_manager.cc:1397] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.103s user 0.024s sys 0.023s
I20250115 14:51:56.402823 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 13c00f03edc74af8b084bb236487a347 (table default.Table_64abfdac91a54d6bb59ccfc7791a170c [id=358f8fad6d944439847ea4ce82f1213a]) successfully deleted
I20250115 14:51:56.403041 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 3a38a6bc32314396b497e7ae166a63f7 (table default.Table_b876bcff1252445ca4b6881550c9dd82 [id=7ffc09afddb142899e28c72544b14b80]) successfully deleted
I20250115 14:51:56.403218 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 5d51e53b4ffa45bfbb9553d7a5870844 (table default.Table_8003836a7a5c4f79a7bec455f2d8a325 [id=817d675e581a4abf848167ce48e46bd0]) successfully deleted
W20250115 14:51:56.403396 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9aeca9975f1945ad956f47db1ea16a73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9aeca9975f1945ad956f47db1ea16a73 already in progress: creating tablet
W20250115 14:51:56.403597 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b92231509c754e8dbfc564813a76a123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b92231509c754e8dbfc564813a76a123 already in progress: creating tablet
W20250115 14:51:56.403769 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet fad6f9f7f853418e85353edfd38cc80e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fad6f9f7f853418e85353edfd38cc80e already in progress: creating tablet
W20250115 14:51:56.403952 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e932d5b820f94130967eccce9e1542fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e932d5b820f94130967eccce9e1542fa already in progress: creating tablet
W20250115 14:51:56.404233 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e50462626f914fb1b0f440c48f30e72d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e50462626f914fb1b0f440c48f30e72d already in progress: creating tablet
W20250115 14:51:56.404436 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 201002us
W20250115 14:51:56.404718 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e932d5b820f94130967eccce9e1542fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e932d5b820f94130967eccce9e1542fa already in progress: creating tablet
I20250115 14:51:56.404979 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d243daba84ff4befa0c4e7bac5fe7476 (table default.Table_c06f4c91e37e4b6f95521174e05efb24 [id=9a6ed225ff3143f5bd2fc0f25324d322]) successfully deleted
W20250115 14:51:56.405200 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 79aad920dd9744d9b87a8234c92232d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79aad920dd9744d9b87a8234c92232d8 already in progress: creating tablet
W20250115 14:51:56.405464 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0de6f6f014074c3d8775cb5e4121ccb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0de6f6f014074c3d8775cb5e4121ccb0 already in progress: creating tablet
W20250115 14:51:56.405691 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0291acf45d75421b8b1c6311560e7f0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0291acf45d75421b8b1c6311560e7f0c already in progress: creating tablet
W20250115 14:51:56.404448 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 79aad920dd9744d9b87a8234c92232d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79aad920dd9744d9b87a8234c92232d8 already in progress: creating tablet
I20250115 14:51:56.406155 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]) successfully deleted
W20250115 14:51:56.406152 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0de6f6f014074c3d8775cb5e4121ccb0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0de6f6f014074c3d8775cb5e4121ccb0 already in progress: creating tablet
I20250115 14:51:56.406769 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e990a2920b542f69a8bdd022db6e7e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4ee8b63cc21145d8993b058d8c6d23c7) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:56.406869 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0291acf45d75421b8b1c6311560e7f0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0291acf45d75421b8b1c6311560e7f0c already in progress: creating tablet
I20250115 14:51:56.407053 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f224f857b1644596853acd9e3a2a4a5f (table default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]) successfully deleted
I20250115 14:51:56.407402 1167 tablet_replica.cc:331] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:56.407389 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e990a2920b542f69a8bdd022db6e7e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4ee8b63cc21145d8993b058d8c6d23c7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.407608 1167 raft_consensus.cc:2238] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.407562 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6e6412160f1746a3a9f165ebe103ee59" } new_table_name: "default.Table_458a573b4a4b4714950723b426f0d265" modify_external_catalogs: true
I20250115 14:51:56.407814 1167 raft_consensus.cc:2267] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.408277 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet bb792ca1b3f54def92f504ec8682d2c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.408591 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_74ce013274db42559be764fbed84c959"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:56.408934 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 5703fc5995184298bf72f9e43efb41f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e6cc08065494d9e85eeff5044b44ba6) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.409404 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 5703fc5995184298bf72f9e43efb41f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e6cc08065494d9e85eeff5044b44ba6) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:56.409616 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 745153c106844076bb020f9ed026c97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.409556 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_39eb08eb474049f2a2fbe741ef730d68" } modify_external_catalogs: true
W20250115 14:51:56.409680 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74ce013274db42559be764fbed84c959 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.409750 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_39eb08eb474049f2a2fbe741ef730d68" } modify_external_catalogs: true
I20250115 14:51:56.410136 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4deac3b7f7124b6aa49514a17811b1c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:56.410511 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcfab1e1f0349739eff636fb1b04806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:56.410616 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4deac3b7f7124b6aa49514a17811b1c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:56.411170 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 3e990a2920b542f69a8bdd022db6e7e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e990a2920b542f69a8bdd022db6e7e2 already in progress: creating tablet
I20250115 14:51:56.399801 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1839edd0d49b467d8bb31b3001ff89eb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.411937 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_01f73e93e1ba462097edfe0e8aaba189"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:56.412536 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d3380e50077a425fbd4ded35b60ba49c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.412966 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3380e50077a425fbd4ded35b60ba49c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.413115 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3c51ba0db5274d3cafe5c89ecf27a771" } new_table_name: "default.Table_b8b42fde86054733ac70ea83d43d6f1b" modify_external_catalogs: true
I20250115 14:51:56.413975 881 tablet_service.cc:1514] Processing DeleteTablet for tablet bb792ca1b3f54def92f504ec8682d2c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:56.414170 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 276abefcab6a422aa6467f00864c01f4 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:56.414608 446 master_service.cc:946] ReplaceTablet: received request to replace tablet f47a8acc3bec46b78725dcc3e675ea74 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:56.414702 1165 tablet_replica.cc:331] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.414899 1165 raft_consensus.cc:2238] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:56.415266 1165 raft_consensus.cc:2267] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:56.416088 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01f73e93e1ba462097edfe0e8aaba189 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.416134 1167 ts_tablet_manager.cc:1905] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.417169 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 259ac764ca844df9b6c0ce1688a188da with delete_type TABLET_DATA_DELETED (replaced by tablet bc8337d181d44ac7bb82551a677b6831) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:56.417237 848 leader_election.cc:343] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.417347 848 leader_election.cc:304] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.417909 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 5027fba8fed34db59b484918256abc45 with delete_type TABLET_DATA_DELETED (replaced by tablet 37ad2b7350294375bd30fb4b46906491) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:56.418007 1594 raft_consensus.cc:2743] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:56.417028 1168 raft_consensus.cc:357] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.418197 1165 ts_tablet_manager.cc:1905] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.418205 1168 raft_consensus.cc:383] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.418290 1168 raft_consensus.cc:738] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.418548 1168 consensus_queue.cc:260] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.419169 1168 ts_tablet_manager.cc:1428] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250115 14:51:56.419952 1168 tablet_bootstrap.cc:492] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:56.421721 1168 tablet_bootstrap.cc:654] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.425374 1167 ts_tablet_manager.cc:1918] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.425549 1167 log.cc:1198] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/3e990a2920b542f69a8bdd022db6e7e2
I20250115 14:51:56.426249 1167 ts_tablet_manager.cc:1939] T 3e990a2920b542f69a8bdd022db6e7e2 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:56.426535 1165 ts_tablet_manager.cc:1918] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:56.426731 1165 log.cc:1198] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/bb792ca1b3f54def92f504ec8682d2c7
I20250115 14:51:56.427155 1165 ts_tablet_manager.cc:1939] T bb792ca1b3f54def92f504ec8682d2c7 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.428506 1165 tablet_replica.cc:331] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.428805 1165 raft_consensus.cc:2238] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:56.429072 1165 pending_rounds.cc:62] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:56.429631 1167 tablet_replica.cc:331] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:56.429903 1167 raft_consensus.cc:2238] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.430083 1167 raft_consensus.cc:2267] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.430642 1167 ts_tablet_manager.cc:1905] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.440502 1166 ts_tablet_manager.cc:1905] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.444368 1165 pending_rounds.cc:69] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558323794309120 op_type: NO_OP noop_request { }
I20250115 14:51:56.444605 1165 raft_consensus.cc:2883] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:56.444813 1165 raft_consensus.cc:2267] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.445108 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet bb792ca1b3f54def92f504ec8682d2c7 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]) successfully deleted
W20250115 14:51:56.445497 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 5703fc5995184298bf72f9e43efb41f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5703fc5995184298bf72f9e43efb41f5 already in progress: creating tablet
I20250115 14:51:56.445727 1165 ts_tablet_manager.cc:1905] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:56.446098 440 catalog_manager.cc:4945] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): alter failed for tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]),no further retry: Not found: Tablet not found: bf909c84eb314d95b73b6de82aebe9f2
I20250115 14:51:56.446811 743 tablet_service.cc:1467] Processing CreateTablet for tablet 009ddfe9dfe14ba2b067f907a9d13538 (DEFAULT_TABLE table=default.Table_872859e534b64345b7989072c4d38b8f [id=816d14738e72406a88b9eed42d72be73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.447223 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009ddfe9dfe14ba2b067f907a9d13538. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.448561 746 tablet_service.cc:1467] Processing CreateTablet for tablet 010e6535b6bc44f391e6f2b6f1686532 (DEFAULT_TABLE table=default.Table_d76649a3901e45f4826621907456750f [id=13d88a401bed452688d5bb6f4dcfd70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.448940 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010e6535b6bc44f391e6f2b6f1686532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.449991 751 tablet_service.cc:1467] Processing CreateTablet for tablet df84bc24a71e431cb47b93f2b0dcaa90 (DEFAULT_TABLE table=default.Table_a9098644161a45d4be352eb0b77e7eca [id=c0c9df829b7c4202b8c36d7d5d767168]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.450320 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df84bc24a71e431cb47b93f2b0dcaa90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.451097 741 tablet_service.cc:1467] Processing CreateTablet for tablet 1f793828f2ca49639c57849afc61ce9e (DEFAULT_TABLE table=default.Table_a4fd4708104e4d52971393e77f803df3 [id=9729a69064894201891fa3f6b37c5aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.451445 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f793828f2ca49639c57849afc61ce9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.451784 455 catalog_manager.cc:5526] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:56.454116 754 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc975b044dd439dbd1f8f9a82cce751 (DEFAULT_TABLE table=default.Table_124bdb85e4194ff5a214a6b79870a1cf [id=f8ea853486b54a39843ef2b5628fbe37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.454418 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc975b044dd439dbd1f8f9a82cce751. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.454767 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 3e990a2920b542f69a8bdd022db6e7e2 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]) successfully deleted
I20250115 14:51:56.456190 1162 tablet_bootstrap.cc:492] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.456367 1162 ts_tablet_manager.cc:1397] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.126s user 0.037s sys 0.038s
I20250115 14:51:56.457122 1162 raft_consensus.cc:357] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.457345 1162 raft_consensus.cc:383] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.457422 1162 raft_consensus.cc:738] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.457669 1162 consensus_queue.cc:260] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.458215 1162 ts_tablet_manager.cc:1428] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:56.465082 1162 tablet_bootstrap.cc:492] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.468478 1165 ts_tablet_manager.cc:1918] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:56.469269 1225 raft_consensus.cc:491] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.469624 1225 raft_consensus.cc:513] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.470989 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25dce523e724066b203e6111cad8111" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:56.471220 913 raft_consensus.cc:2463] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:56.473474 983 leader_election.cc:304] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.473750 1231 raft_consensus.cc:2798] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.475581 1231 raft_consensus.cc:491] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.475642 1162 tablet_bootstrap.cc:654] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.475709 1231 raft_consensus.cc:3054] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.476091 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25dce523e724066b203e6111cad8111" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:56.477078 984 leader_election.cc:343] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.478169 1594 raft_consensus.cc:491] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.478317 1594 raft_consensus.cc:513] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.479009 1594 leader_election.cc:290] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.479426 1225 leader_election.cc:290] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.476893 1165 log.cc:1198] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/745153c106844076bb020f9ed026c97a
I20250115 14:51:56.486099 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f0144904ae4008945ccf325039dac1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.486367 1047 raft_consensus.cc:2463] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:56.486825 847 leader_election.cc:304] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.487169 1594 raft_consensus.cc:2798] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.487273 1594 raft_consensus.cc:491] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.487360 1594 raft_consensus.cc:3054] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.488327 1168 tablet_bootstrap.cc:492] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.488507 1168 ts_tablet_manager.cc:1397] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.006s
I20250115 14:51:56.490010 1594 raft_consensus.cc:513] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.491288 1168 raft_consensus.cc:357] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.491422 1594 leader_election.cc:290] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.491503 1168 raft_consensus.cc:383] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.491593 1168 raft_consensus.cc:738] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.491808 1168 consensus_queue.cc:260] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.492233 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f0144904ae4008945ccf325039dac1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:56.492378 1168 ts_tablet_manager.cc:1428] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:51:56.492449 1047 raft_consensus.cc:3054] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:56.492719 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 5703fc5995184298bf72f9e43efb41f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5703fc5995184298bf72f9e43efb41f5 already in progress: creating tablet
I20250115 14:51:56.494112 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc975b044dd439dbd1f8f9a82cce751 (DEFAULT_TABLE table=default.Table_124bdb85e4194ff5a214a6b79870a1cf [id=f8ea853486b54a39843ef2b5628fbe37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.494247 1164 tablet_bootstrap.cc:492] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:56.494395 1164 ts_tablet_manager.cc:1397] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.105s user 0.001s sys 0.004s
I20250115 14:51:56.494503 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc975b044dd439dbd1f8f9a82cce751. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.495015 1047 raft_consensus.cc:2463] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:56.495127 1164 raft_consensus.cc:357] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.495323 1164 raft_consensus.cc:383] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.495400 1164 raft_consensus.cc:738] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.495433 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f0144904ae4008945ccf325039dac1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.495632 1164 consensus_queue.cc:260] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.495803 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86f0144904ae4008945ccf325039dac1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:56.496124 1164 ts_tablet_manager.cc:1428] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:56.496105 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4821224ac8f1472d8b1e8afc06a118a3 (table default.Table_721928c380b840ad84f9f5030b52632f [id=c5f831177088497dbf1fee130e231e37]). Expected version 1 got 0
I20250115 14:51:56.496471 1164 tablet_bootstrap.cc:492] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:56.496543 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 86f0144904ae4008945ccf325039dac1 (table default.Table_39eb08eb474049f2a2fbe741ef730d68 [id=ab6f49066e884652b77a0b9a791575c7]). Expected version 1 got 0
I20250115 14:51:56.496579 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 009ddfe9dfe14ba2b067f907a9d13538 (DEFAULT_TABLE table=default.Table_872859e534b64345b7989072c4d38b8f [id=816d14738e72406a88b9eed42d72be73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.496948 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009ddfe9dfe14ba2b067f907a9d13538. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.496770 455 catalog_manager.cc:5526] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:56.497212 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]). Expected version 2 got 0
I20250115 14:51:56.497499 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet ab1a38186c5042ff851f57979ab80dbc (table default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]). Expected version 2 got 0
I20250115 14:51:56.498041 1164 tablet_bootstrap.cc:654] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.500516 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 010e6535b6bc44f391e6f2b6f1686532 (DEFAULT_TABLE table=default.Table_d76649a3901e45f4826621907456750f [id=13d88a401bed452688d5bb6f4dcfd70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.503005 1166 ts_tablet_manager.cc:1918] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:56.503134 1166 log.cc:1198] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/bf909c84eb314d95b73b6de82aebe9f2
I20250115 14:51:56.503538 1166 ts_tablet_manager.cc:1939] T bf909c84eb314d95b73b6de82aebe9f2 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:56.508662 879 tablet_service.cc:1467] Processing CreateTablet for tablet 1f793828f2ca49639c57849afc61ce9e (DEFAULT_TABLE table=default.Table_a4fd4708104e4d52971393e77f803df3 [id=9729a69064894201891fa3f6b37c5aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.509014 881 tablet_service.cc:1467] Processing CreateTablet for tablet 009ddfe9dfe14ba2b067f907a9d13538 (DEFAULT_TABLE table=default.Table_872859e534b64345b7989072c4d38b8f [id=816d14738e72406a88b9eed42d72be73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.509347 1468 raft_consensus.cc:491] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.509487 1468 raft_consensus.cc:513] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.510224 878 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc975b044dd439dbd1f8f9a82cce751 (DEFAULT_TABLE table=default.Table_124bdb85e4194ff5a214a6b79870a1cf [id=f8ea853486b54a39843ef2b5628fbe37]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.510596 1468 leader_election.cc:290] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.510892 888 tablet_service.cc:1467] Processing CreateTablet for tablet df84bc24a71e431cb47b93f2b0dcaa90 (DEFAULT_TABLE table=default.Table_a9098644161a45d4be352eb0b77e7eca [id=c0c9df829b7c4202b8c36d7d5d767168]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.511204 847 leader_election.cc:304] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.511665 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f793828f2ca49639c57849afc61ce9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.508740 890 tablet_service.cc:1467] Processing CreateTablet for tablet 010e6535b6bc44f391e6f2b6f1686532 (DEFAULT_TABLE table=default.Table_d76649a3901e45f4826621907456750f [id=13d88a401bed452688d5bb6f4dcfd70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.512118 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df84bc24a71e431cb47b93f2b0dcaa90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.512924 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010e6535b6bc44f391e6f2b6f1686532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.513929 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 009ddfe9dfe14ba2b067f907a9d13538. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.514951 1021 tablet_service.cc:1467] Processing CreateTablet for tablet df84bc24a71e431cb47b93f2b0dcaa90 (DEFAULT_TABLE table=default.Table_a9098644161a45d4be352eb0b77e7eca [id=c0c9df829b7c4202b8c36d7d5d767168]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.516824 1468 raft_consensus.cc:2798] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.517946 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bc975b044dd439dbd1f8f9a82cce751. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:56.517968 848 leader_election.cc:343] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.518466 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 010e6535b6bc44f391e6f2b6f1686532. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.522442 1209 raft_consensus.cc:491] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.522625 1209 raft_consensus.cc:513] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.523280 1209 leader_election.cc:290] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.524535 1468 raft_consensus.cc:695] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:56.524828 1468 consensus_queue.cc:237] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.525841 1165 ts_tablet_manager.cc:1939] T 745153c106844076bb020f9ed026c97a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.527119 882 tablet_service.cc:1467] Processing CreateTablet for tablet be63bae96c834a8bbbb1174c027935a2 (DEFAULT_TABLE table=default.Table_ad45cc2411ad4ee1a65b9db7a0a6e4b5 [id=b2c6ee1ec113494cac7d1bf11a928714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.527236 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet bf909c84eb314d95b73b6de82aebe9f2 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]) successfully deleted
W20250115 14:51:56.527469 848 leader_election.cc:343] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.527725 887 tablet_service.cc:1467] Processing CreateTablet for tablet 104282a09c024bbaafeb5f28f91dadcb (DEFAULT_TABLE table=default.Table_23f12c7168a84d5390b6d4ad9ed07ea1 [id=09205f90c7fc476185a1d31a60ad5fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.527979 886 tablet_service.cc:1467] Processing CreateTablet for tablet d71a1ecd900942e5ae788c9a28c70a7a (DEFAULT_TABLE table=default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.528230 885 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed (DEFAULT_TABLE table=default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 [id=cdf45f410fda46f29cd6dbfd1cc45f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.528586 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86075ae5b894d5986ac4bd538f4d42a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.528736 883 tablet_service.cc:1467] Processing CreateTablet for tablet 4b68c168048649c5b91600eb4302b0ce (DEFAULT_TABLE table=default.Table_a1bd89ddf322401092a05b3f20970798 [id=30b975fd37374a5a90d8cbda734489f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.529143 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a63a0a8bb44fe9be0740fc777e610c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.529188 892 tablet_service.cc:1467] Processing CreateTablet for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 (DEFAULT_TABLE table=default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.529618 884 tablet_service.cc:1467] Processing CreateTablet for tablet 7d07686c40324233a49e4c49aeed73f6 (DEFAULT_TABLE table=default.Table_8f5c258dbdc04de79751c3a1b995e0fb [id=e71b0f487dc0428aaa03b4cc7cf95aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.529861 751 tablet_service.cc:1467] Processing CreateTablet for tablet d71a1ecd900942e5ae788c9a28c70a7a (DEFAULT_TABLE table=default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.527491 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be63bae96c834a8bbbb1174c027935a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.530239 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d71a1ecd900942e5ae788c9a28c70a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.530539 880 tablet_service.cc:1467] Processing CreateTablet for tablet 338c0f6f39fd4335bbcda733b28b8bf8 (DEFAULT_TABLE table=default.Table_c14780544d7c4f43899713097a27848c [id=6756cde43edd4d1e8a2768c101c4aa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.530853 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 338c0f6f39fd4335bbcda733b28b8bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.531903 1231 raft_consensus.cc:513] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.532588 1231 leader_election.cc:290] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.533072 746 tablet_service.cc:1467] Processing CreateTablet for tablet be63bae96c834a8bbbb1174c027935a2 (DEFAULT_TABLE table=default.Table_ad45cc2411ad4ee1a65b9db7a0a6e4b5 [id=b2c6ee1ec113494cac7d1bf11a928714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.533154 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 1f793828f2ca49639c57849afc61ce9e (DEFAULT_TABLE table=default.Table_a4fd4708104e4d52971393e77f803df3 [id=9729a69064894201891fa3f6b37c5aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.533413 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be63bae96c834a8bbbb1174c027935a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.533633 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25dce523e724066b203e6111cad8111" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:56.534229 984 leader_election.cc:343] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.535025 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df84bc24a71e431cb47b93f2b0dcaa90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.535224 743 tablet_service.cc:1467] Processing CreateTablet for tablet 4b68c168048649c5b91600eb4302b0ce (DEFAULT_TABLE table=default.Table_a1bd89ddf322401092a05b3f20970798 [id=30b975fd37374a5a90d8cbda734489f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.535564 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b68c168048649c5b91600eb4302b0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.536335 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4b68c168048649c5b91600eb4302b0ce (DEFAULT_TABLE table=default.Table_a1bd89ddf322401092a05b3f20970798 [id=30b975fd37374a5a90d8cbda734489f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.536864 1020 tablet_service.cc:1467] Processing CreateTablet for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 (DEFAULT_TABLE table=default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.537082 1014 tablet_service.cc:1467] Processing CreateTablet for tablet d71a1ecd900942e5ae788c9a28c70a7a (DEFAULT_TABLE table=default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.537286 741 tablet_service.cc:1467] Processing CreateTablet for tablet 104282a09c024bbaafeb5f28f91dadcb (DEFAULT_TABLE table=default.Table_23f12c7168a84d5390b6d4ad9ed07ea1 [id=09205f90c7fc476185a1d31a60ad5fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.537377 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 7d07686c40324233a49e4c49aeed73f6 (DEFAULT_TABLE table=default.Table_8f5c258dbdc04de79751c3a1b995e0fb [id=e71b0f487dc0428aaa03b4cc7cf95aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.537549 1019 tablet_service.cc:1467] Processing CreateTablet for tablet be63bae96c834a8bbbb1174c027935a2 (DEFAULT_TABLE table=default.Table_ad45cc2411ad4ee1a65b9db7a0a6e4b5 [id=b2c6ee1ec113494cac7d1bf11a928714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.537616 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104282a09c024bbaafeb5f28f91dadcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.537833 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 338c0f6f39fd4335bbcda733b28b8bf8 (DEFAULT_TABLE table=default.Table_c14780544d7c4f43899713097a27848c [id=6756cde43edd4d1e8a2768c101c4aa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.538040 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 104282a09c024bbaafeb5f28f91dadcb (DEFAULT_TABLE table=default.Table_23f12c7168a84d5390b6d4ad9ed07ea1 [id=09205f90c7fc476185a1d31a60ad5fa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.538305 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 3e94ea76507f4646909ac0c1b2c3136a (DEFAULT_TABLE table=default.Table_bbb6df0135694d7b83ed96983e487001 [id=3903b654e8644b2ab335a5f98f106261]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.538484 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed (DEFAULT_TABLE table=default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 [id=cdf45f410fda46f29cd6dbfd1cc45f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.538911 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (DEFAULT_TABLE table=default.Table_70c7c9d33ea742618f8a32464ed92f8e [id=471b029f1dba4dd087deb339bba02013]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.539337 1018 tablet_service.cc:1467] Processing CreateTablet for tablet caa1827366a34f039fd12deb855f07b6 (DEFAULT_TABLE table=default.Table_a43e4255e5c3414aa47bff8bbe111d13 [id=aa88c2d281fe4856b6d6af91c9359f23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.539785 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 (DEFAULT_TABLE table=default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.534914 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a63a0a8bb44fe9be0740fc777e610c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.541337 1048 raft_consensus.cc:2463] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:56.541590 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f793828f2ca49639c57849afc61ce9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.541865 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d07686c40324233a49e4c49aeed73f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.542865 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d8c53fdf16243f0a614e48ec4ac94f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.543114 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 338c0f6f39fd4335bbcda733b28b8bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.544061 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed (DEFAULT_TABLE table=default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 [id=cdf45f410fda46f29cd6dbfd1cc45f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.544404 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.526432 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86075ae5b894d5986ac4bd538f4d42a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.545889 756 tablet_service.cc:1467] Processing CreateTablet for tablet 7d07686c40324233a49e4c49aeed73f6 (DEFAULT_TABLE table=default.Table_8f5c258dbdc04de79751c3a1b995e0fb [id=e71b0f487dc0428aaa03b4cc7cf95aa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.546073 743 tablet_service.cc:1467] Processing CreateTablet for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 (DEFAULT_TABLE table=default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.546222 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d07686c40324233a49e4c49aeed73f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.546429 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.546725 752 tablet_service.cc:1467] Processing CreateTablet for tablet 338c0f6f39fd4335bbcda733b28b8bf8 (DEFAULT_TABLE table=default.Table_c14780544d7c4f43899713097a27848c [id=6756cde43edd4d1e8a2768c101c4aa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.547049 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 338c0f6f39fd4335bbcda733b28b8bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.548594 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e94ea76507f4646909ac0c1b2c3136a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.548594 744 tablet_service.cc:1467] Processing CreateTablet for tablet 3e94ea76507f4646909ac0c1b2c3136a (DEFAULT_TABLE table=default.Table_bbb6df0135694d7b83ed96983e487001 [id=3903b654e8644b2ab335a5f98f106261]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.548935 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e94ea76507f4646909ac0c1b2c3136a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.551287 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 745153c106844076bb020f9ed026c97a (table default.Table_031dcfe40eda4ff49338ab79a0cf9844 [id=9cda5b5b1bcf47dfbc5b5a8dcc69b44a]) successfully deleted
I20250115 14:51:56.552137 740 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (DEFAULT_TABLE table=default.Table_70c7c9d33ea742618f8a32464ed92f8e [id=471b029f1dba4dd087deb339bba02013]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.552556 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.555747 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 05d590c63d6e4d058fac50dc50010317 (DEFAULT_TABLE table=default.Table_67aaf987cc30461bb07c319c25639ffa [id=ef96f8cc27264b11803d888505942ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.557204 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.557351 847 leader_election.cc:304] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.557638 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d07686c40324233a49e4c49aeed73f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.557094 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 0e924821d12045f6bc658461675aead4 (DEFAULT_TABLE table=default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.558382 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3e94ea76507f4646909ac0c1b2c3136a (DEFAULT_TABLE table=default.Table_bbb6df0135694d7b83ed96983e487001 [id=3903b654e8644b2ab335a5f98f106261]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.558564 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104282a09c024bbaafeb5f28f91dadcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.559054 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b25dce523e724066b203e6111cad8111" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:56.559273 913 raft_consensus.cc:3054] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.560459 1168 tablet_bootstrap.cc:492] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:56.565040 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d71a1ecd900942e5ae788c9a28c70a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.567126 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d71a1ecd900942e5ae788c9a28c70a7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.571877 1594 raft_consensus.cc:2798] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.573243 1594 raft_consensus.cc:491] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250115 14:51:56.574539 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f847f92cd122423aa93f1afa701b0bbb consensus state has no leader
I20250115 14:51:56.577973 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.578224 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.578153 889 tablet_service.cc:1467] Processing CreateTablet for tablet caa1827366a34f039fd12deb855f07b6 (DEFAULT_TABLE table=default.Table_a43e4255e5c3414aa47bff8bbe111d13 [id=aa88c2d281fe4856b6d6af91c9359f23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.578153 893 tablet_service.cc:1467] Processing CreateTablet for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (DEFAULT_TABLE table=default.Table_70c7c9d33ea742618f8a32464ed92f8e [id=471b029f1dba4dd087deb339bba02013]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.580345 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 104282a09c024bbaafeb5f28f91dadcb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:56.581053 847 leader_election.cc:343] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.582739 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be63bae96c834a8bbbb1174c027935a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.583495 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d590c63d6e4d058fac50dc50010317. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.584520 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e924821d12045f6bc658461675aead4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.585260 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa1827366a34f039fd12deb855f07b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.579803 1594 raft_consensus.cc:3054] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.585635 877 tablet_service.cc:1467] Processing CreateTablet for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 (DEFAULT_TABLE table=default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.586151 1025 tablet_service.cc:1467] Processing CreateTablet for tablet c6f9262e808b4589a21257ed3fb0363b (DEFAULT_TABLE table=default.Table_02294519a7244bd6a9f4741db4a92d58 [id=eed10013d4554afbb09538836b5d3164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.586671 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b68c168048649c5b91600eb4302b0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.586973 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f9262e808b4589a21257ed3fb0363b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.587090 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b68c168048649c5b91600eb4302b0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.588096 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.589340 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.589907 879 tablet_service.cc:1467] Processing CreateTablet for tablet 0e924821d12045f6bc658461675aead4 (DEFAULT_TABLE table=default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.590176 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa1827366a34f039fd12deb855f07b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.590368 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e924821d12045f6bc658461675aead4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.591050 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e94ea76507f4646909ac0c1b2c3136a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.591262 776 raft_consensus.cc:3054] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.592872 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.609632 890 tablet_service.cc:1467] Processing CreateTablet for tablet 05d590c63d6e4d058fac50dc50010317 (DEFAULT_TABLE table=default.Table_67aaf987cc30461bb07c319c25639ffa [id=ef96f8cc27264b11803d888505942ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.610561 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5d8548b30284461086fdcd351454ad57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.611131 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d8548b30284461086fdcd351454ad57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.613046 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d98a5978bcea4e01af0c694f42d2308a" } new_table_name: "default.Table_ca74aa81918e4ee08ddb4801b0cbff16" modify_external_catalogs: true
I20250115 14:51:56.614518 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 23cba5cc51d7414a9352caedcc362c9c
I20250115 14:51:56.615180 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_843fb099548148f29e7b622a4ad34b61" } modify_external_catalogs: true
I20250115 14:51:56.615326 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_843fb099548148f29e7b622a4ad34b61" } modify_external_catalogs: true
I20250115 14:51:56.615892 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f47a8acc3bec46b78725dcc3e675ea74
I20250115 14:51:56.616338 446 catalog_manager.cc:6596] ReplaceTablet: tablet f47a8acc3bec46b78725dcc3e675ea74 deleted and replaced by tablet ea8ab6a34caa486ab929af2cac3ce957
I20250115 14:51:56.610087 876 tablet_service.cc:1467] Processing CreateTablet for tablet c6f9262e808b4589a21257ed3fb0363b (DEFAULT_TABLE table=default.Table_02294519a7244bd6a9f4741db4a92d58 [id=eed10013d4554afbb09538836b5d3164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.616495 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d8c53fdf16243f0a614e48ec4ac94f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.616634 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2a410c48ef764ba2ac337b8471f11ba1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.617143 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a410c48ef764ba2ac337b8471f11ba1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.618038 889 tablet_service.cc:1514] Processing DeleteTablet for tablet f47a8acc3bec46b78725dcc3e675ea74 with delete_type TABLET_DATA_DELETED (replaced by tablet ea8ab6a34caa486ab929af2cac3ce957) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:56.618264 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_121cfa7da22441d581d821791e428785" } modify_external_catalogs: true
I20250115 14:51:56.618428 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_121cfa7da22441d581d821791e428785" } modify_external_catalogs: true
I20250115 14:51:56.621133 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cba5cc51d7414a9352caedcc362c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:56.622694 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d70c898088424b64902d7ba0be41017d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.623174 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d70c898088424b64902d7ba0be41017d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:56.625988 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 consensus state has no leader
I20250115 14:51:56.628479 913 raft_consensus.cc:2463] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:56.628651 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_874111e557c1400bb8fb21c733becad1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.629171 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_874111e557c1400bb8fb21c733becad1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.630399 1162 tablet_bootstrap.cc:492] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.631273 983 leader_election.cc:304] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.633045 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cba5cc51d7414a9352caedcc362c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.636590 1162 ts_tablet_manager.cc:1397] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.171s user 0.002s sys 0.024s
I20250115 14:51:56.637676 750 tablet_service.cc:1467] Processing CreateTablet for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 (DEFAULT_TABLE table=default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.638860 746 tablet_service.cc:1467] Processing CreateTablet for tablet caa1827366a34f039fd12deb855f07b6 (DEFAULT_TABLE table=default.Table_a43e4255e5c3414aa47bff8bbe111d13 [id=aa88c2d281fe4856b6d6af91c9359f23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.640759 1231 raft_consensus.cc:2798] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.642043 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d590c63d6e4d058fac50dc50010317. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:56.642339 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 23cba5cc51d7414a9352caedcc362c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23cba5cc51d7414a9352caedcc362c9c already in progress: creating tablet
I20250115 14:51:56.643520 454 catalog_manager.cc:5526] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:56.643914 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4b90205ef15f470a99c013e7f2cf99c6 (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]). Expected version 3 got 0
I20250115 14:51:56.644330 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:51:56.644531 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 1839edd0d49b467d8bb31b3001ff89eb (table default.Table_253ac9c181914d0f891baf1f8722dbd0 [id=a724f6e4ad16485b8ee2445bb072be3e]). Expected version 1 got 0
I20250115 14:51:56.645805 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet f47a8acc3bec46b78725dcc3e675ea74 with delete_type TABLET_DATA_DELETED (replaced by tablet ea8ab6a34caa486ab929af2cac3ce957) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.645921 1231 raft_consensus.cc:695] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:56.649155 1231 consensus_queue.cc:237] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.650183 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d8c53fdf16243f0a614e48ec4ac94f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:56.651866 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f47a8acc3bec46b78725dcc3e675ea74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47a8acc3bec46b78725dcc3e675ea74 already in progress: creating tablet
I20250115 14:51:56.652127 749 tablet_service.cc:1467] Processing CreateTablet for tablet 0e924821d12045f6bc658461675aead4 (DEFAULT_TABLE table=default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.653679 1468 raft_consensus.cc:491] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.659188 748 tablet_service.cc:1467] Processing CreateTablet for tablet 05d590c63d6e4d058fac50dc50010317 (DEFAULT_TABLE table=default.Table_67aaf987cc30461bb07c319c25639ffa [id=ef96f8cc27264b11803d888505942ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.665004 741 tablet_service.cc:1467] Processing CreateTablet for tablet c6f9262e808b4589a21257ed3fb0363b (DEFAULT_TABLE table=default.Table_02294519a7244bd6a9f4741db4a92d58 [id=eed10013d4554afbb09538836b5d3164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:56.665380 1164 tablet_bootstrap.cc:492] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:56.665262 1166 tablet_replica.cc:331] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:56.665519 1164 ts_tablet_manager.cc:1397] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.169s user 0.006s sys 0.001s
I20250115 14:51:56.666282 1164 raft_consensus.cc:357] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.666472 1164 raft_consensus.cc:383] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.666558 1164 raft_consensus.cc:738] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.666759 1164 consensus_queue.cc:260] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.667240 1164 ts_tablet_manager.cc:1428] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:56.668041 1594 raft_consensus.cc:513] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.668921 1468 raft_consensus.cc:513] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.669883 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cba5cc51d7414a9352caedcc362c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.670023 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caa1827366a34f039fd12deb855f07b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.670547 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e924821d12045f6bc658461675aead4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.672698 1162 raft_consensus.cc:357] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.673091 1162 raft_consensus.cc:383] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.677235 450 master_service.cc:946] ReplaceTablet: received request to replace tablet ab1a38186c5042ff851f57979ab80dbc from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:56.679235 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a63a0a8bb44fe9be0740fc777e610c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:56.679448 1047 raft_consensus.cc:3054] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.682574 1468 leader_election.cc:290] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.683254 1225 raft_consensus.cc:491] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.685027 1167 ts_tablet_manager.cc:1918] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.685288 1162 raft_consensus.cc:738] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.685951 1594 leader_election.cc:290] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.686156 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f9262e808b4589a21257ed3fb0363b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.687928 1162 consensus_queue.cc:260] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.688607 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a5a92804cb462181e68e1cec02102b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.689810 1162 ts_tablet_manager.cc:1428] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.052s user 0.000s sys 0.002s
I20250115 14:51:56.690101 1162 tablet_bootstrap.cc:492] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.689127 1225 raft_consensus.cc:513] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.696177 1589 raft_consensus.cc:491] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.696354 1589 raft_consensus.cc:513] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:56.696597 847 leader_election.cc:343] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.697028 1589 leader_election.cc:290] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.702420 1168 tablet_bootstrap.cc:654] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.703179 1167 log.cc:1198] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9bcfab1e1f0349739eff636fb1b04806
I20250115 14:51:56.703800 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685a04e7454f6592f6ab83c356f466" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.704437 1225 leader_election.cc:290] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.705153 1167 ts_tablet_manager.cc:1939] T 9bcfab1e1f0349739eff636fb1b04806 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:56.705924 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9bcfab1e1f0349739eff636fb1b04806 (table default.Table_fa872457db7140b6a13bfc998e32581b [id=4e1c43b0e14844babf0bb0cf8032a3fa]) successfully deleted
I20250115 14:51:56.706411 1167 tablet_replica.cc:331] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:56.706624 1167 raft_consensus.cc:2238] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
W20250115 14:51:56.707120 984 leader_election.cc:343] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.710567 1047 raft_consensus.cc:2463] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:56.711340 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685a04e7454f6592f6ab83c356f466" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:56.712167 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05d590c63d6e4d058fac50dc50010317. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:56.712749 1164 tablet_bootstrap.cc:492] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:56.712744 847 leader_election.cc:304] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.713021 1594 raft_consensus.cc:491] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.713316 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6f9262e808b4589a21257ed3fb0363b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:56.720046 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3e990a2920b542f69a8bdd022db6e7e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e990a2920b542f69a8bdd022db6e7e2 already in progress: creating tablet
W20250115 14:51:56.720615 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet bb792ca1b3f54def92f504ec8682d2c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bb792ca1b3f54def92f504ec8682d2c7 already in progress: creating tablet
W20250115 14:51:56.720819 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5703fc5995184298bf72f9e43efb41f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5703fc5995184298bf72f9e43efb41f5 already in progress: creating tablet
I20250115 14:51:56.721274 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4161fce81834cd2ba4ffb789f6a5ba3" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.723501 1162 tablet_bootstrap.cc:654] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.726397 743 tablet_service.cc:1514] Processing DeleteTablet for tablet f47a8acc3bec46b78725dcc3e675ea74 with delete_type TABLET_DATA_DELETED (replaced by tablet ea8ab6a34caa486ab929af2cac3ce957) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.726658 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a63a0a8bb44fe9be0740fc777e610c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:56.726848 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a5a92804cb462181e68e1cec02102b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.732420 1167 raft_consensus.cc:2267] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.733206 1594 raft_consensus.cc:513] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.733920 1594 leader_election.cc:290] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.734170 913 raft_consensus.cc:2463] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:56.734625 983 leader_election.cc:304] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.735123 1231 raft_consensus.cc:2798] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.735242 1231 raft_consensus.cc:491] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.735322 1231 raft_consensus.cc:3054] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.737756 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d43cd66b18f4649894a629637037822 with delete_type TABLET_DATA_DELETED (replaced by tablet bb792ca1b3f54def92f504ec8682d2c7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.738122 1231 raft_consensus.cc:513] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.738739 1231 leader_election.cc:290] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.739562 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeca9975f1945ad956f47db1ea16a73" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:56.740319 1468 raft_consensus.cc:2798] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:51:56.741674 848 leader_election.cc:343] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.741887 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685a04e7454f6592f6ab83c356f466" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:56.743073 984 leader_election.cc:343] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.745527 848 leader_election.cc:304] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.747310 1167 ts_tablet_manager.cc:1905] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.747726 1468 raft_consensus.cc:695] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:56.747998 1468 consensus_queue.cc:237] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.748416 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeca9975f1945ad956f47db1ea16a73" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:56.749154 847 leader_election.cc:343] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:56.750152 848 leader_election.cc:343] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:56.750480 848 leader_election.cc:343] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:56.750749 848 leader_election.cc:343] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.750842 848 leader_election.cc:304] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.751129 1209 raft_consensus.cc:2743] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:56.751307 1209 raft_consensus.cc:2743] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:56.753161 1162 tablet_bootstrap.cc:492] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.753335 1162 ts_tablet_manager.cc:1397] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.063s user 0.017s sys 0.014s
I20250115 14:51:56.754081 1162 raft_consensus.cc:357] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.754299 1162 raft_consensus.cc:383] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.754379 1162 raft_consensus.cc:738] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.754637 1162 consensus_queue.cc:260] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.755142 1162 ts_tablet_manager.cc:1428] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:56.760792 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4161fce81834cd2ba4ffb789f6a5ba3" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:56.761296 1162 tablet_bootstrap.cc:492] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.763087 1162 tablet_bootstrap.cc:654] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.760249 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36685a04e7454f6592f6ab83c356f466" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:56.764652 913 raft_consensus.cc:3054] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.774106 913 raft_consensus.cc:2463] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:56.774638 983 leader_election.cc:304] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.775101 1231 raft_consensus.cc:2798] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.775266 1231 raft_consensus.cc:695] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:56.775565 1231 consensus_queue.cc:237] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.775856 710 leader_election.cc:304] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:56.778245 1164 tablet_bootstrap.cc:654] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.804073 1167 ts_tablet_manager.cc:1918] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250115 14:51:56.808188 984 consensus_peers.cc:487] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:56.808493 1166 raft_consensus.cc:2238] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:51:56.808949 1589 raft_consensus.cc:668] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopping
I20250115 14:51:56.809207 1589 raft_consensus.cc:2743] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:56.810786 1231 consensus_queue.cc:1035] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:56.825515 1162 tablet_bootstrap.cc:492] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.825703 1162 ts_tablet_manager.cc:1397] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.064s user 0.014s sys 0.037s
I20250115 14:51:56.826551 1162 raft_consensus.cc:357] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.826798 1162 raft_consensus.cc:383] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.826885 1162 raft_consensus.cc:738] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.827112 1162 consensus_queue.cc:260] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.827661 1162 ts_tablet_manager.cc:1428] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:56.827978 1162 tablet_bootstrap.cc:492] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.829815 1162 tablet_bootstrap.cc:654] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.832036 1164 tablet_bootstrap.cc:492] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:56.832193 1164 ts_tablet_manager.cc:1397] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.120s user 0.007s sys 0.004s
W20250115 14:51:56.830799 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 69322.9us
I20250115 14:51:56.833528 1633 tablet.cc:1721] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:51:56.837462 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 171643c40c9b43ffb823f7c33789284d consensus state has no leader
I20250115 14:51:56.837973 1540 raft_consensus.cc:491] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.809631 1166 raft_consensus.cc:2267] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.838127 1540 raft_consensus.cc:513] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.838774 1540 leader_election.cc:290] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.839197 452 master_service.cc:946] ReplaceTablet: received request to replace tablet b2c385edf4414157845b971aaf366831 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:56.841248 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1839edd0d49b467d8bb31b3001ff89eb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:56.841485 909 raft_consensus.cc:2463] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:56.841971 983 leader_election.cc:304] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:56.842406 1540 raft_consensus.cc:2798] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.842500 1540 raft_consensus.cc:491] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.842592 1540 raft_consensus.cc:3054] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.844950 1231 raft_consensus.cc:491] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.845083 1231 raft_consensus.cc:513] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.845322 1540 raft_consensus.cc:513] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.845865 1540 leader_election.cc:290] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.846771 1231 leader_election.cc:290] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.847442 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1839edd0d49b467d8bb31b3001ff89eb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:56.847489 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a5a92804cb462181e68e1cec02102b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:56.847685 910 raft_consensus.cc:2463] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
W20250115 14:51:56.847858 984 consensus_peers.cc:487] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:56.847690 909 raft_consensus.cc:3054] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.847824 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1839edd0d49b467d8bb31b3001ff89eb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.847996 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a5a92804cb462181e68e1cec02102b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.848214 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1839edd0d49b467d8bb31b3001ff89eb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:56.848404 984 leader_election.cc:343] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:56.848726 984 leader_election.cc:343] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:56.848935 984 leader_election.cc:343] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.849261 983 leader_election.cc:304] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.849498 1231 raft_consensus.cc:2798] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:56.849611 1166 ts_tablet_manager.cc:1905] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.849640 1231 raft_consensus.cc:491] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:56.849987 1231 raft_consensus.cc:3054] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.850651 909 raft_consensus.cc:2463] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:56.851145 983 leader_election.cc:304] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.851550 1540 raft_consensus.cc:2798] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.851887 1540 raft_consensus.cc:695] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:56.852207 1540 consensus_queue.cc:237] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.853551 1231 raft_consensus.cc:513] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.854596 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a5a92804cb462181e68e1cec02102b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:56.854923 1594 raft_consensus.cc:491] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.855058 1594 raft_consensus.cc:513] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:56.855659 1594 leader_election.cc:290] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.856129 1164 raft_consensus.cc:357] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.856241 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50edca843e54f4b8e803d4d180b736d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:56.856751 847 leader_election.cc:343] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:56.858871 984 leader_election.cc:343] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.860106 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a5a92804cb462181e68e1cec02102b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:56.860451 909 raft_consensus.cc:3054] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:56.863219 909 raft_consensus.cc:2463] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:56.863381 1231 leader_election.cc:290] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.865998 983 leader_election.cc:304] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.868067 1231 raft_consensus.cc:2798] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:56.868456 1231 raft_consensus.cc:695] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:56.868978 1231 consensus_queue.cc:237] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.871876 1167 log.cc:1198] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/bb792ca1b3f54def92f504ec8682d2c7
I20250115 14:51:56.872440 1167 ts_tablet_manager.cc:1939] T bb792ca1b3f54def92f504ec8682d2c7 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:56.873014 1167 tablet_replica.cc:331] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:56.873297 1167 raft_consensus.cc:2238] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:56.874109 1167 pending_rounds.cc:62] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:56.874250 1167 pending_rounds.cc:69] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558325500428288 op_type: NO_OP noop_request { }
I20250115 14:51:56.874419 1167 raft_consensus.cc:2883] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:56.874583 1167 raft_consensus.cc:2267] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.875414 1167 ts_tablet_manager.cc:1905] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:56.900542 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 149784us
W20250115 14:51:56.900974 848 leader_election.cc:343] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.901206 848 leader_election.cc:304] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:56.904681 1594 raft_consensus.cc:2743] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:56.907130 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet bb792ca1b3f54def92f504ec8682d2c7 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]) successfully deleted
I20250115 14:51:56.907891 1594 raft_consensus.cc:491] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.908041 1594 raft_consensus.cc:513] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.908726 1594 leader_election.cc:290] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.909339 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83097c5bfe364418a213236435edc40c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:56.909889 847 leader_election.cc:343] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.919270 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d97c2f443ab040f9bc1924f45379165e
I20250115 14:51:56.919896 448 master_service.cc:946] ReplaceTablet: received request to replace tablet f847f92cd122423aa93f1afa701b0bbb from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:56.921077 1162 tablet_bootstrap.cc:492] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:56.921244 1162 ts_tablet_manager.cc:1397] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.093s user 0.029s sys 0.026s
I20250115 14:51:56.922039 1162 raft_consensus.cc:357] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.922259 1162 raft_consensus.cc:383] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.922325 1162 raft_consensus.cc:738] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.922535 1162 consensus_queue.cc:260] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.923030 1162 ts_tablet_manager.cc:1428] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:56.923307 1162 tablet_bootstrap.cc:492] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:56.925509 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_db31d0a74f474c2c8d12130b143b1371"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:56.925822 1162 tablet_bootstrap.cc:654] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:56.926013 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db31d0a74f474c2c8d12130b143b1371 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.926688 1231 raft_consensus.cc:491] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.926849 1231 raft_consensus.cc:513] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.929852 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_05f796ba50b14c1f973f89c85c133164"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.930328 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05f796ba50b14c1f973f89c85c133164 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.931448 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 86f0144904ae4008945ccf325039dac1
I20250115 14:51:56.931505 1231 leader_election.cc:290] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:56.932137 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c14780544d7c4f43899713097a27848c" } new_table_name: "default.Table_c230050c7eab42e7a66a8132c7421def" modify_external_catalogs: true
I20250115 14:51:56.934316 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet d97c2f443ab040f9bc1924f45379165e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.934605 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 86f0144904ae4008945ccf325039dac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.934813 1165 tablet_replica.cc:331] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.935032 1165 raft_consensus.cc:2238] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:56.935196 1165 raft_consensus.cc:2267] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:56.935729 1165 ts_tablet_manager.cc:1905] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:56.939822 1628 raft_consensus.cc:491] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:56.940544 1164 raft_consensus.cc:383] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:56.942034 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1e7580b6245b41628874d9dcbc3366e2
I20250115 14:51:56.942826 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ab1a38186c5042ff851f57979ab80dbc
I20250115 14:51:56.943214 450 catalog_manager.cc:6596] ReplaceTablet: tablet ab1a38186c5042ff851f57979ab80dbc deleted and replaced by tablet 07a000ab73dd46ac8e134784a156040a
I20250115 14:51:56.943488 1165 ts_tablet_manager.cc:1918] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.943620 1165 log.cc:1198] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/86f0144904ae4008945ccf325039dac1
W20250115 14:51:56.944340 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d97c2f443ab040f9bc1924f45379165e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d97c2f443ab040f9bc1924f45379165e already in progress: creating tablet
I20250115 14:51:56.944461 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1a38186c5042ff851f57979ab80dbc with delete_type TABLET_DATA_DELETED (replaced by tablet 07a000ab73dd46ac8e134784a156040a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.943948 1165 ts_tablet_manager.cc:1939] T 86f0144904ae4008945ccf325039dac1 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:56.944650 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]). Expected version 2 got 0
I20250115 14:51:56.944898 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet e0eba208f9be4a588cc81e4d99336eac (table default.Table_2898e18fe4e64a1a92ed3658b91dc502 [id=6d0e848982e245e1870c661f7dc35138]). Expected version 1 got 0
I20250115 14:51:56.945315 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7] alter complete (version 2)
I20250115 14:51:56.946041 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4239747374aa4d6d866961430786677c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.946749 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50edca843e54f4b8e803d4d180b736d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.947194 750 tablet_service.cc:1514] Processing DeleteTablet for tablet d97c2f443ab040f9bc1924f45379165e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.947337 1164 raft_consensus.cc:738] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:56.947669 1164 consensus_queue.cc:260] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.947789 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b2c385edf4414157845b971aaf366831
I20250115 14:51:56.948202 452 catalog_manager.cc:6596] ReplaceTablet: tablet b2c385edf4414157845b971aaf366831 deleted and replaced by tablet 84844b5f50af408ab7dda73f7421cd9d
I20250115 14:51:56.948204 1164 ts_tablet_manager.cc:1428] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.116s user 0.001s sys 0.001s
I20250115 14:51:56.948307 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 86f0144904ae4008945ccf325039dac1 (table default.Table_39eb08eb474049f2a2fbe741ef730d68 [id=ab6f49066e884652b77a0b9a791575c7]) successfully deleted
I20250115 14:51:56.948835 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7580b6245b41628874d9dcbc3366e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.949590 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet fad6f9f7f853418e85353edfd38cc80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.949687 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 259ac764ca844df9b6c0ce1688a188da with delete_type TABLET_DATA_DELETED (replaced by tablet bc8337d181d44ac7bb82551a677b6831) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.949813 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f327ffa731c4c949b24a34dc54ae260) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.949882 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet bb792ca1b3f54def92f504ec8682d2c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.950038 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 745153c106844076bb020f9ed026c97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.950234 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c385edf4414157845b971aaf366831 with delete_type TABLET_DATA_DELETED (replaced by tablet 84844b5f50af408ab7dda73f7421cd9d) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.950425 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 5027fba8fed34db59b484918256abc45 with delete_type TABLET_DATA_DELETED (replaced by tablet 37ad2b7350294375bd30fb4b46906491) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:56.950649 984 leader_election.cc:343] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:56.940027 1628 raft_consensus.cc:513] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:56.951970 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcfab1e1f0349739eff636fb1b04806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.953603 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 86f0144904ae4008945ccf325039dac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.956498 1628 leader_election.cc:290] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:56.956820 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83097c5bfe364418a213236435edc40c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:56.957114 1164 tablet_bootstrap.cc:492] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:56.958961 750 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1a38186c5042ff851f57979ab80dbc with delete_type TABLET_DATA_DELETED (replaced by tablet 07a000ab73dd46ac8e134784a156040a) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.959422 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f847f92cd122423aa93f1afa701b0bbb
I20250115 14:51:56.959863 448 catalog_manager.cc:6596] ReplaceTablet: tablet f847f92cd122423aa93f1afa701b0bbb deleted and replaced by tablet 83415f1c07ad495cabd287bdcd54dad4
I20250115 14:51:56.960716 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3d00e26e980d4ef7aa3517cbaedb6fca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:56.961174 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbe00fc656b44633ae2fd159ae77ffc0" } modify_external_catalogs: true
W20250115 14:51:56.961292 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d00e26e980d4ef7aa3517cbaedb6fca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.961352 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbe00fc656b44633ae2fd159ae77ffc0" } modify_external_catalogs: true
I20250115 14:51:56.961864 1164 tablet_bootstrap.cc:654] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:56.962339 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7580b6245b41628874d9dcbc3366e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.962726 750 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c385edf4414157845b971aaf366831 with delete_type TABLET_DATA_DELETED (replaced by tablet 84844b5f50af408ab7dda73f7421cd9d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.963995 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8f5c258dbdc04de79751c3a1b995e0fb" } modify_external_catalogs: true
I20250115 14:51:56.964201 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8f5c258dbdc04de79751c3a1b995e0fb" } modify_external_catalogs: true
I20250115 14:51:56.964473 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a1bd89ddf322401092a05b3f20970798" } new_table_name: "default.Table_8d0846f2d6464fdd98780567261d93f3" modify_external_catalogs: true
I20250115 14:51:56.965512 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet f847f92cd122423aa93f1afa701b0bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 83415f1c07ad495cabd287bdcd54dad4) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:56.965942 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_27c15993e89541219ace5c7251953112" } new_table_name: "default.Table_5bb88f06405a4a91b7deda4c03d9ba5f" modify_external_catalogs: true
W20250115 14:51:56.968598 710 leader_election.cc:343] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 8d43cd66b18f4649894a629637037822
I20250115 14:51:56.968740 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_950b48d1b1344479a7229041f044bbb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.969182 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_950b48d1b1344479a7229041f044bbb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.969314 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f847f92cd122423aa93f1afa701b0bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 83415f1c07ad495cabd287bdcd54dad4) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.969501 750 tablet_service.cc:1514] Processing DeleteTablet for tablet f6205e20a1984ccabccc45b93f9c7fa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:56.985421 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 86f0144904ae4008945ccf325039dac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:56.985642 1168 tablet_bootstrap.cc:492] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:56.985845 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7580b6245b41628874d9dcbc3366e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:56.985930 1165 tablet_replica.cc:331] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:56.986135 1165 raft_consensus.cc:2238] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:56.986404 1165 pending_rounds.cc:62] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:56.986469 1165 pending_rounds.cc:69] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558325666398208 op_type: NO_OP noop_request { }
I20250115 14:51:56.986608 1165 raft_consensus.cc:2883] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:56.990677 453 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet f960c88df14242c5ada1ecf9d8e68196 (table default.Table_56a365aae805419683d1f4dfd97951cc [id=24d4e203bce64fcdbd5b6fa30a99e6b2]). Expected version 1 got 0
I20250115 14:51:56.991662 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8e068b869e304185abaccf2d63685cd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.992184 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e068b869e304185abaccf2d63685cd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.994894 1166 ts_tablet_manager.cc:1918] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:56.995040 1166 log.cc:1198] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/74bfda3861d148bc8b67ea071909d22d
I20250115 14:51:56.995354 1166 ts_tablet_manager.cc:1939] T 74bfda3861d148bc8b67ea071909d22d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:51:56.996528 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 338c0f6f39fd4335bbcda733b28b8bf8 consensus state has no leader
I20250115 14:51:56.996877 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9c06e47841fa45c59c310505e26c9361" } new_table_name: "default.Table_7488025a06ba42b5814d9788890cf2a0" modify_external_catalogs: true
I20250115 14:51:56.997130 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_00dac00a165544449ce9b8dff8efab98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.997633 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00dac00a165544449ce9b8dff8efab98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.996238 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:56.998508 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:56.999744 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 74bfda3861d148bc8b67ea071909d22d (table default.Table_a78c10f9e6384d7d89d3add6f1371429 [id=2c59ffcdee944c4289ad5dd38dc2e8cc]) successfully deleted
I20250115 14:51:57.000706 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet f90e4edc7aea43268d581b441b788f27
I20250115 14:51:57.000990 447 master_service.cc:946] ReplaceTablet: received request to replace tablet e7100e6a51ed4e178e86138f10d5ffc2 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:57.003460 1165 raft_consensus.cc:2267] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.005834 879 tablet_service.cc:1514] Processing DeleteTablet for tablet d97c2f443ab040f9bc1924f45379165e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.006803 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a63a0a8bb44fe9be0740fc777e610c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.007041 887 tablet_service.cc:1514] Processing DeleteTablet for tablet e50462626f914fb1b0f440c48f30e72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.007176 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aeca9975f1945ad956f47db1ea16a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:57.007495 710 leader_election.cc:343] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 8d43cd66b18f4649894a629637037822
I20250115 14:51:57.007639 710 leader_election.cc:304] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:57.008116 848 leader_election.cc:343] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.008725 776 raft_consensus.cc:3054] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.009596 1628 raft_consensus.cc:2743] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:57.010510 1165 ts_tablet_manager.cc:1905] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.014106 1578 catalog_manager.cc:1260] Loaded cluster ID: 838a9b5bdaf340e2b93e4cbb7e53a2a4
I20250115 14:51:57.014220 1578 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: loading cluster ID for follower catalog manager: success
W20250115 14:51:57.015422 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d71a1ecd900942e5ae788c9a28c70a7a (table default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d71a1ecd900942e5ae788c9a28c70a7a consensus state has no leader
W20250115 14:51:57.017918 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b68c168048649c5b91600eb4302b0ce consensus state has no leader
I20250115 14:51:57.022897 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 36685a04e7454f6592f6ab83c356f466 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:57.023427 890 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c385edf4414157845b971aaf366831 with delete_type TABLET_DATA_DELETED (replaced by tablet 84844b5f50af408ab7dda73f7421cd9d) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.023461 884 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1a38186c5042ff851f57979ab80dbc with delete_type TABLET_DATA_DELETED (replaced by tablet 07a000ab73dd46ac8e134784a156040a) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:57.023895 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59d8fd4b26294e8ca6745b95f2e03231 consensus state has no leader
W20250115 14:51:57.026044 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e198b4e1a65b4704bf2bb8f7d5987385 consensus state has no leader
I20250115 14:51:57.026109 1633 tablet.cc:1721] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:57.030306 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]). Expected version 2 got 0
I20250115 14:51:57.033267 1164 tablet_bootstrap.cc:492] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.033233 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]). Expected version 2 got 1
I20250115 14:51:57.033493 1164 ts_tablet_manager.cc:1397] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.001s
I20250115 14:51:57.034672 1164 raft_consensus.cc:357] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.034958 1164 raft_consensus.cc:383] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.035100 1164 raft_consensus.cc:738] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.035171 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b25dce523e724066b203e6111cad8111 with delete_type TABLET_DATA_DELETED (replaced by tablet 0de6f6f014074c3d8775cb5e4121ccb0) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.035454 1168 ts_tablet_manager.cc:1397] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.453s user 0.002s sys 0.012s
I20250115 14:51:57.035400 1164 consensus_queue.cc:260] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.035849 1578 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: acquiring CA information for follower catalog manager: success
I20250115 14:51:57.036135 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f847f92cd122423aa93f1afa701b0bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 83415f1c07ad495cabd287bdcd54dad4) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.036722 1468 raft_consensus.cc:491] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.036867 1468 raft_consensus.cc:513] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.038496 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b60ff80679a470cb4bdced71cc14195" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:57.040659 750 tablet_service.cc:1514] Processing DeleteTablet for tablet ad67354f35ce4564b88dbdffcabf3564 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.043759 1165 ts_tablet_manager.cc:1918] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:57.043912 1165 log.cc:1198] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ab1a38186c5042ff851f57979ab80dbc
I20250115 14:51:57.044342 1165 ts_tablet_manager.cc:1939] T ab1a38186c5042ff851f57979ab80dbc P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.045328 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ab1a38186c5042ff851f57979ab80dbc (table default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]) successfully deleted
I20250115 14:51:57.046671 1578 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250115 14:51:57.046955 1165 tablet_replica.cc:331] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.047370 1162 tablet_bootstrap.cc:492] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:57.047503 1162 ts_tablet_manager.cc:1397] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.124s user 0.028s sys 0.039s
I20250115 14:51:57.047488 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c5ab1b94f5894ae084dc0539c56b0487" } new_table_name: "default.Table_83a9122be22e48a5ad8b22b98c838001" modify_external_catalogs: true
I20250115 14:51:57.048228 1162 raft_consensus.cc:357] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.048435 1162 raft_consensus.cc:383] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.048506 1162 raft_consensus.cc:738] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.048710 1162 consensus_queue.cc:260] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.049183 1162 ts_tablet_manager.cc:1428] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:57.049453 1162 tablet_bootstrap.cc:492] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:57.050834 1468 leader_election.cc:290] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:57.051191 1162 tablet_bootstrap.cc:654] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.051299 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]). Expected version 2 got 1
I20250115 14:51:57.052241 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e990a2920b542f69a8bdd022db6e7e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4ee8b63cc21145d8993b058d8c6d23c7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.052680 1633 tablet.cc:1721] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:57.049412 1165 raft_consensus.cc:2238] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:57.053058 1165 pending_rounds.cc:62] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:57.053339 1164 ts_tablet_manager.cc:1428] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:51:57.053339 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4239747374aa4d6d866961430786677c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:57.053643 1164 tablet_bootstrap.cc:492] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.053673 913 raft_consensus.cc:2463] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:57.053656 1165 pending_rounds.cc:69] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558325765357568 op_type: NO_OP noop_request { }
I20250115 14:51:57.053800 1165 raft_consensus.cc:2883] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:57.053893 1165 raft_consensus.cc:2267] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.054139 983 leader_election.cc:304] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:57.054461 847 leader_election.cc:343] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.055215 1165 ts_tablet_manager.cc:1905] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.055846 1225 raft_consensus.cc:2798] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:57.055959 1225 raft_consensus.cc:491] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:57.056044 1225 raft_consensus.cc:3054] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.058861 1231 raft_consensus.cc:491] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.059235 1231 raft_consensus.cc:513] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.059873 1231 leader_election.cc:290] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:57.060829 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 23cba5cc51d7414a9352caedcc362c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23cba5cc51d7414a9352caedcc362c9c already in progress: creating tablet
W20250115 14:51:57.061071 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f47a8acc3bec46b78725dcc3e675ea74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47a8acc3bec46b78725dcc3e675ea74 already in progress: creating tablet
I20250115 14:51:57.058632 1225 raft_consensus.cc:513] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.063514 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4239747374aa4d6d866961430786677c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:57.064054 984 leader_election.cc:343] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.064445 1166 tablet_replica.cc:331] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.064667 1166 raft_consensus.cc:2238] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.064769 1225 leader_election.cc:290] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.065347 1165 ts_tablet_manager.cc:1918] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:57.065377 1166 raft_consensus.cc:2267] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.065208 1164 tablet_bootstrap.cc:654] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.068017 1165 log.cc:1198] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/1e7580b6245b41628874d9dcbc3366e2
I20250115 14:51:57.068075 876 tablet_service.cc:1514] Processing DeleteTablet for tablet d86075ae5b894d5986ac4bd538f4d42a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.068586 1165 ts_tablet_manager.cc:1939] T 1e7580b6245b41628874d9dcbc3366e2 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.069217 877 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f327ffa731c4c949b24a34dc54ae260) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.069396 848 leader_election.cc:304] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:57.069368 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fcacad621d8e46eebb110daa55789690"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.069880 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcacad621d8e46eebb110daa55789690 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.070137 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4239747374aa4d6d866961430786677c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:57.070318 910 raft_consensus.cc:3054] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.070858 1468 raft_consensus.cc:491] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.071101 1468 raft_consensus.cc:513] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.071288 1165 tablet_replica.cc:331] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.071345 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 1e7580b6245b41628874d9dcbc3366e2 (table default.Table_843fb099548148f29e7b622a4ad34b61 [id=085d5f79fc6c4bdb87752ecc55d89445]) successfully deleted
I20250115 14:51:57.071463 1165 raft_consensus.cc:2238] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250115 14:51:57.071594 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet fad6f9f7f853418e85353edfd38cc80e because the tablet was not found. No further retry: Not found: Tablet not found: fad6f9f7f853418e85353edfd38cc80e
I20250115 14:51:57.071614 1165 raft_consensus.cc:2267] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.071679 1468 leader_election.cc:290] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:57.072088 1165 ts_tablet_manager.cc:1905] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.068217 1166 ts_tablet_manager.cc:1905] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.073915 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5027fba8fed34db59b484918256abc45" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:57.074105 913 raft_consensus.cc:2463] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:57.072384 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50462626f914fb1b0f440c48f30e72d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:57.074517 1047 raft_consensus.cc:2463] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:57.075119 983 leader_election.cc:304] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:57.053066 1168 raft_consensus.cc:357] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.076344 847 leader_election.cc:304] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:57.076673 1468 raft_consensus.cc:2798] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:57.076766 1468 raft_consensus.cc:491] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:57.076855 1468 raft_consensus.cc:3054] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.078739 1165 ts_tablet_manager.cc:1918] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:57.079068 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3a7a487a42949bd9641c38d6c1fcfd3 consensus state has no leader
I20250115 14:51:57.079308 1628 raft_consensus.cc:491] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.079459 1628 raft_consensus.cc:513] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.080051 1628 leader_election.cc:290] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:57.082397 750 tablet_service.cc:1467] Processing CreateTablet for tablet 9853dcdfee5f43de84798cfa0bef674e (DEFAULT_TABLE table=default.Table_4ec686b46a094818bd820c4f3854220b [id=5dc3e53ee73a4441960dc231d02d163f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.082834 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9853dcdfee5f43de84798cfa0bef674e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.080897 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f960c88df14242c5ada1ecf9d8e68196" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:57.084283 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9853dcdfee5f43de84798cfa0bef674e (DEFAULT_TABLE table=default.Table_4ec686b46a094818bd820c4f3854220b [id=5dc3e53ee73a4441960dc231d02d163f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.084983 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9853dcdfee5f43de84798cfa0bef674e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.085214 1165 log.cc:1198] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/259ac764ca844df9b6c0ce1688a188da
I20250115 14:51:57.085594 1165 ts_tablet_manager.cc:1939] T 259ac764ca844df9b6c0ce1688a188da P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.087790 1165 tablet_replica.cc:331] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.088025 1165 raft_consensus.cc:2238] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.088194 1165 raft_consensus.cc:2267] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.088430 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 259ac764ca844df9b6c0ce1688a188da (table default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]) successfully deleted
I20250115 14:51:57.088694 1225 raft_consensus.cc:2798] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:57.088860 1225 raft_consensus.cc:491] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:57.089038 1225 raft_consensus.cc:3054] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.090632 1231 raft_consensus.cc:491] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.090782 1231 raft_consensus.cc:513] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.091393 1231 leader_election.cc:290] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.091831 1225 raft_consensus.cc:513] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.092129 748 tablet_service.cc:1467] Processing CreateTablet for tablet 4abde784f9c948cb96e6cc93c30817bd (DEFAULT_TABLE table=default.Table_10deaf92a667422fa029a731f755dfef [id=e657e05ca35045f1a196dd012ee59483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.092567 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4abde784f9c948cb96e6cc93c30817bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.096166 1166 ts_tablet_manager.cc:1918] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:57.096575 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 88457.3us
I20250115 14:51:57.096855 1168 raft_consensus.cc:383] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.096940 1168 raft_consensus.cc:738] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.097105 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 50273a0019bb4cd8977f7ef014372d85 (DEFAULT_TABLE table=default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.097456 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50273a0019bb4cd8977f7ef014372d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.099948 1624 raft_consensus.cc:2743] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:57.100152 848 leader_election.cc:343] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.100263 848 leader_election.cc:304] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:57.100536 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d9ce65604c294021af93632ed073f627 (DEFAULT_TABLE table=default.Table_79fecf866fd6405a9b7df222877fc2b0 [id=0748f5a86b034a96b1dfe1cd723d2438]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.100731 1594 raft_consensus.cc:2743] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:57.100896 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ce65604c294021af93632ed073f627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.101747 886 tablet_service.cc:1467] Processing CreateTablet for tablet 01291d41a2b345b4bfd00e6be0fb0129 (DEFAULT_TABLE table=default.Table_21fa1a9206d64c8bb19a405f37518a65 [id=e31104604e8648c08fd706e44150cda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.101982 890 tablet_service.cc:1467] Processing CreateTablet for tablet 759a4ebe7b0444128ddd31a78aad463e (DEFAULT_TABLE table=default.Table_d1345e860a054b708e11f2f64347a3d7 [id=87e2283ba18a4525ab55a8316238aac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.102141 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01291d41a2b345b4bfd00e6be0fb0129. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.102229 877 tablet_service.cc:1467] Processing CreateTablet for tablet 9853dcdfee5f43de84798cfa0bef674e (DEFAULT_TABLE table=default.Table_4ec686b46a094818bd820c4f3854220b [id=5dc3e53ee73a4441960dc231d02d163f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.102370 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 759a4ebe7b0444128ddd31a78aad463e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.102573 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9853dcdfee5f43de84798cfa0bef674e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.102608 884 tablet_service.cc:1467] Processing CreateTablet for tablet f3b75d759f834d558c94cc981e338ac0 (DEFAULT_TABLE table=default.Table_01bf4fd686f74e258977d847e2ee966e [id=f4f61992d591495f9e7a5a4b5c94fa32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.101742 892 tablet_service.cc:1467] Processing CreateTablet for tablet d9ce65604c294021af93632ed073f627 (DEFAULT_TABLE table=default.Table_79fecf866fd6405a9b7df222877fc2b0 [id=0748f5a86b034a96b1dfe1cd723d2438]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.103699 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5027fba8fed34db59b484918256abc45" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:57.104318 876 tablet_service.cc:1467] Processing CreateTablet for tablet 4abde784f9c948cb96e6cc93c30817bd (DEFAULT_TABLE table=default.Table_10deaf92a667422fa029a731f755dfef [id=e657e05ca35045f1a196dd012ee59483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.104586 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 01291d41a2b345b4bfd00e6be0fb0129 (DEFAULT_TABLE table=default.Table_21fa1a9206d64c8bb19a405f37518a65 [id=e31104604e8648c08fd706e44150cda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.104866 1225 leader_election.cc:290] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.104988 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01291d41a2b345b4bfd00e6be0fb0129. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.088762 1165 ts_tablet_manager.cc:1905] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.105070 1168 consensus_queue.cc:260] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.109823 1468 raft_consensus.cc:513] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.110683 883 tablet_service.cc:1467] Processing CreateTablet for tablet 50273a0019bb4cd8977f7ef014372d85 (DEFAULT_TABLE table=default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.110848 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50462626f914fb1b0f440c48f30e72d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:57.111045 1047 raft_consensus.cc:3054] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.111615 1016 tablet_service.cc:1467] Processing CreateTablet for tablet f3b75d759f834d558c94cc981e338ac0 (DEFAULT_TABLE table=default.Table_01bf4fd686f74e258977d847e2ee966e [id=f4f61992d591495f9e7a5a4b5c94fa32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.111968 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3b75d759f834d558c94cc981e338ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.113687 1047 raft_consensus.cc:2463] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:57.114142 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 67e842e1c2b44754b3b9a2e026486a84 (DEFAULT_TABLE table=default.Table_21621e400c244baab7e990067052f131 [id=9b4d541a96b34ce5a567f3e0982c72c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.114472 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e842e1c2b44754b3b9a2e026486a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.115160 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 759a4ebe7b0444128ddd31a78aad463e (DEFAULT_TABLE table=default.Table_d1345e860a054b708e11f2f64347a3d7 [id=87e2283ba18a4525ab55a8316238aac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.115507 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 759a4ebe7b0444128ddd31a78aad463e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.120546 1012 tablet_service.cc:1467] Processing CreateTablet for tablet ea8ab6a34caa486ab929af2cac3ce957 (DEFAULT_TABLE table=default.Table_aa6a106222694bbfbe6c26b0f8e65396 [id=8725c85d30684db495ec7a20962fd75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.120705 910 raft_consensus.cc:2463] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:57.120883 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea8ab6a34caa486ab929af2cac3ce957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.121150 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 36685a04e7454f6592f6ab83c356f466
I20250115 14:51:57.121609 453 catalog_manager.cc:6596] ReplaceTablet: tablet 36685a04e7454f6592f6ab83c356f466 deleted and replaced by tablet f06544890cbf460cbe218690e998eb1d
I20250115 14:51:57.121935 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7d07686c40324233a49e4c49aeed73f6
I20250115 14:51:57.122601 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e7100e6a51ed4e178e86138f10d5ffc2
I20250115 14:51:57.122967 447 catalog_manager.cc:6596] ReplaceTablet: tablet e7100e6a51ed4e178e86138f10d5ffc2 deleted and replaced by tablet 6942d102069d4c33b23fa2c0b29f94a6
I20250115 14:51:57.123131 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3b75d759f834d558c94cc981e338ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.124030 847 leader_election.cc:304] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:57.124115 448 catalog_manager.cc:5526] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:57.124739 448 catalog_manager.cc:5526] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:57.125176 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 1839edd0d49b467d8bb31b3001ff89eb (table default.Table_253ac9c181914d0f891baf1f8722dbd0 [id=a724f6e4ad16485b8ee2445bb072be3e]). Expected version 1 got 0
I20250115 14:51:57.125427 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4239747374aa4d6d866961430786677c (table default.Table_d810e40dae3d42e38f38ebf32e62305e [id=8941c6c27037475990cf318614854a27]). Expected version 1 got 0
I20250115 14:51:57.125602 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]). Expected version 2 got 0
I20250115 14:51:57.127228 983 leader_election.cc:304] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:57.127442 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_992f266d1ab24ee68d346afbfdec91eb" } modify_external_catalogs: true
I20250115 14:51:57.127609 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_992f266d1ab24ee68d346afbfdec91eb" } modify_external_catalogs: true
I20250115 14:51:57.128136 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet e7100e6a51ed4e178e86138f10d5ffc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6942d102069d4c33b23fa2c0b29f94a6) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.128300 1168 ts_tablet_manager.cc:1428] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.086s user 0.002s sys 0.000s
I20250115 14:51:57.128427 879 tablet_service.cc:1467] Processing CreateTablet for tablet ea8ab6a34caa486ab929af2cac3ce957 (DEFAULT_TABLE table=default.Table_aa6a106222694bbfbe6c26b0f8e65396 [id=8725c85d30684db495ec7a20962fd75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.128541 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f960c88df14242c5ada1ecf9d8e68196" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:51:57.129216 984 leader_election.cc:343] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.129348 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7ec0e7f2e4de4b14bd6cea8489658118"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:57.129765 1225 raft_consensus.cc:2798] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:51:57.129860 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ec0e7f2e4de4b14bd6cea8489658118 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.129918 1225 raft_consensus.cc:695] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:57.130154 1225 consensus_queue.cc:237] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.130316 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 36685a04e7454f6592f6ab83c356f466 with delete_type TABLET_DATA_DELETED (replaced by tablet f06544890cbf460cbe218690e998eb1d) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.130462 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07686c40324233a49e4c49aeed73f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.130594 887 tablet_service.cc:1467] Processing CreateTablet for tablet 67e842e1c2b44754b3b9a2e026486a84 (DEFAULT_TABLE table=default.Table_21621e400c244baab7e990067052f131 [id=9b4d541a96b34ce5a567f3e0982c72c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.132259 1468 leader_election.cc:290] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:57.133002 1594 raft_consensus.cc:2798] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:57.133139 1594 raft_consensus.cc:695] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:57.134791 1594 consensus_queue.cc:237] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.135346 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ef39f0505cbe4de9b46458d25eafb9e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.135793 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef39f0505cbe4de9b46458d25eafb9e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.136359 885 tablet_service.cc:1514] Processing DeleteTablet for tablet e7100e6a51ed4e178e86138f10d5ffc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6942d102069d4c33b23fa2c0b29f94a6) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.137187 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0826262d07e14081b5a0ab88d9d4a20f" } modify_external_catalogs: true
I20250115 14:51:57.137373 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0826262d07e14081b5a0ab88d9d4a20f" } modify_external_catalogs: true
I20250115 14:51:57.139827 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4abde784f9c948cb96e6cc93c30817bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.139901 1168 tablet_bootstrap.cc:492] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:57.141541 1168 tablet_bootstrap.cc:654] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.142354 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07686c40324233a49e4c49aeed73f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.142462 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e842e1c2b44754b3b9a2e026486a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.142541 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 36685a04e7454f6592f6ab83c356f466 with delete_type TABLET_DATA_DELETED (replaced by tablet f06544890cbf460cbe218690e998eb1d) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.142876 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 4abde784f9c948cb96e6cc93c30817bd (DEFAULT_TABLE table=default.Table_10deaf92a667422fa029a731f755dfef [id=e657e05ca35045f1a196dd012ee59483]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.143211 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4abde784f9c948cb96e6cc93c30817bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.143430 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ce65604c294021af93632ed073f627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.146013 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50273a0019bb4cd8977f7ef014372d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.146795 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea8ab6a34caa486ab929af2cac3ce957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.153091 1628 raft_consensus.cc:491] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.154912 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeca9975f1945ad956f47db1ea16a73" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:57.155071 1166 log.cc:1198] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/8d43cd66b18f4649894a629637037822
I20250115 14:51:57.155336 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8" } modify_external_catalogs: true
I20250115 14:51:57.155499 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8" } modify_external_catalogs: true
I20250115 14:51:57.155838 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5027fba8fed34db59b484918256abc45" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:57.155861 1628 raft_consensus.cc:513] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.156563 1165 ts_tablet_manager.cc:1918] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.156700 1165 log.cc:1198] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d4fb2d33f3c745beb4a4ba036e23a8e0
I20250115 14:51:57.148342 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5027fba8fed34db59b484918256abc45" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
W20250115 14:51:57.156762 984 leader_election.cc:343] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.159219 1165 ts_tablet_manager.cc:1939] T d4fb2d33f3c745beb4a4ba036e23a8e0 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.161754 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d4fb2d33f3c745beb4a4ba036e23a8e0 (table default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]) successfully deleted
W20250115 14:51:57.162575 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet bb792ca1b3f54def92f504ec8682d2c7 because the tablet was not found. No further retry: Not found: Tablet not found: bb792ca1b3f54def92f504ec8682d2c7
W20250115 14:51:57.163089 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 745153c106844076bb020f9ed026c97a because the tablet was not found. No further retry: Not found: Tablet not found: 745153c106844076bb020f9ed026c97a
W20250115 14:51:57.163697 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b2c385edf4414157845b971aaf366831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2c385edf4414157845b971aaf366831 already in progress: creating tablet
I20250115 14:51:57.163930 1628 leader_election.cc:290] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:57.164032 1165 tablet_replica.cc:331] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.164208 1165 raft_consensus.cc:2238] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.164819 1165 raft_consensus.cc:2267] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.165555 1165 ts_tablet_manager.cc:1905] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.165627 1162 tablet_bootstrap.cc:492] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:57.167284 1162 ts_tablet_manager.cc:1397] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.118s user 0.022s sys 0.001s
I20250115 14:51:57.167552 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50462626f914fb1b0f440c48f30e72d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:57.168478 984 consensus_peers.cc:487] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:57.168902 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b60ff80679a470cb4bdced71cc14195" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:57.169251 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeca9975f1945ad956f47db1ea16a73" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:57.169462 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50462626f914fb1b0f440c48f30e72d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:57.169751 984 leader_election.cc:343] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.170318 750 tablet_service.cc:1467] Processing CreateTablet for tablet 50273a0019bb4cd8977f7ef014372d85 (DEFAULT_TABLE table=default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.170651 749 tablet_service.cc:1467] Processing CreateTablet for tablet 01291d41a2b345b4bfd00e6be0fb0129 (DEFAULT_TABLE table=default.Table_21fa1a9206d64c8bb19a405f37518a65 [id=e31104604e8648c08fd706e44150cda1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.171049 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50273a0019bb4cd8977f7ef014372d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.172165 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01291d41a2b345b4bfd00e6be0fb0129. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.173409 710 leader_election.cc:304] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:57.173976 740 tablet_service.cc:1467] Processing CreateTablet for tablet ea8ab6a34caa486ab929af2cac3ce957 (DEFAULT_TABLE table=default.Table_aa6a106222694bbfbe6c26b0f8e65396 [id=8725c85d30684db495ec7a20962fd75d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.174396 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07686c40324233a49e4c49aeed73f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.175966 746 tablet_service.cc:1467] Processing CreateTablet for tablet d9ce65604c294021af93632ed073f627 (DEFAULT_TABLE table=default.Table_79fecf866fd6405a9b7df222877fc2b0 [id=0748f5a86b034a96b1dfe1cd723d2438]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.175863 741 tablet_service.cc:1467] Processing CreateTablet for tablet f3b75d759f834d558c94cc981e338ac0 (DEFAULT_TABLE table=default.Table_01bf4fd686f74e258977d847e2ee966e [id=f4f61992d591495f9e7a5a4b5c94fa32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.177604 756 tablet_service.cc:1467] Processing CreateTablet for tablet 67e842e1c2b44754b3b9a2e026486a84 (DEFAULT_TABLE table=default.Table_21621e400c244baab7e990067052f131 [id=9b4d541a96b34ce5a567f3e0982c72c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.177928 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 37ad2b7350294375bd30fb4b46906491
I20250115 14:51:57.179132 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 629873c00550408798b00489ece4ea5f
I20250115 14:51:57.179833 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 629873c00550408798b00489ece4ea5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.181774 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ad2b7350294375bd30fb4b46906491 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.182194 1653 tablet.cc:1721] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:57.182932 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ad2b7350294375bd30fb4b46906491 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.183044 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 629873c00550408798b00489ece4ea5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.184229 752 tablet_service.cc:1467] Processing CreateTablet for tablet 759a4ebe7b0444128ddd31a78aad463e (DEFAULT_TABLE table=default.Table_d1345e860a054b708e11f2f64347a3d7 [id=87e2283ba18a4525ab55a8316238aac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.184376 744 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 (DEFAULT_TABLE table=default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.184592 1166 ts_tablet_manager.cc:1939] T 8d43cd66b18f4649894a629637037822 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.184736 754 tablet_service.cc:1514] Processing DeleteTablet for tablet e7100e6a51ed4e178e86138f10d5ffc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6942d102069d4c33b23fa2c0b29f94a6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.184727 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 6b60ff80679a470cb4bdced71cc14195 (table default.Table_c088519ba27c457983f341363f8a109e [id=201c214e2eab4873af2095e83dc88b23]). Expected version 2 got 0
I20250115 14:51:57.184934 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]). Expected version 2 got 0
I20250115 14:51:57.185006 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea8ab6a34caa486ab929af2cac3ce957. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.185226 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 759a4ebe7b0444128ddd31a78aad463e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.185416 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 36685a04e7454f6592f6ab83c356f466 with delete_type TABLET_DATA_DELETED (replaced by tablet f06544890cbf460cbe218690e998eb1d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.185712 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6cc08065494d9e85eeff5044b44ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.188421 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ce65604c294021af93632ed073f627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.188982 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1a38186c5042ff851f57979ab80dbc with delete_type TABLET_DATA_DELETED (replaced by tablet 07a000ab73dd46ac8e134784a156040a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.189186 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e50462626f914fb1b0f440c48f30e72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.189344 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet b25dce523e724066b203e6111cad8111 with delete_type TABLET_DATA_DELETED (replaced by tablet 0de6f6f014074c3d8775cb5e4121ccb0) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.189478 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7580b6245b41628874d9dcbc3366e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.189607 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aeca9975f1945ad956f47db1ea16a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.192364 1628 raft_consensus.cc:2743] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:57.192428 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3b75d759f834d558c94cc981e338ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.196414 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67e842e1c2b44754b3b9a2e026486a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.199133 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a63a0a8bb44fe9be0740fc777e610c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.202147 1165 ts_tablet_manager.cc:1918] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.204348 1165 log.cc:1198] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/5027fba8fed34db59b484918256abc45
I20250115 14:51:57.208261 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7b4c900318f642499ecffcb268290262"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.208842 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b4c900318f642499ecffcb268290262 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.209213 739 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee8b63cc21145d8993b058d8c6d23c7 (DEFAULT_TABLE table=default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.209568 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee8b63cc21145d8993b058d8c6d23c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.209517 751 tablet_service.cc:1467] Processing CreateTablet for tablet ec2a8000f5474f62aae52c8e88e45276 (DEFAULT_TABLE table=default.Table_01f73e93e1ba462097edfe0e8aaba189 [id=1b0b109197b0474395a710dd740f944f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.209862 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec2a8000f5474f62aae52c8e88e45276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.210108 743 tablet_service.cc:1467] Processing CreateTablet for tablet 43605c995df4400e91139199c9514bfd (DEFAULT_TABLE table=default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.210217 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3d00e26e980d4ef7aa3517cbaedb6fca" } modify_external_catalogs: true
I20250115 14:51:57.210371 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3d00e26e980d4ef7aa3517cbaedb6fca" } modify_external_catalogs: true
I20250115 14:51:57.210464 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43605c995df4400e91139199c9514bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.211406 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1699f914f5e94be1b121f649a762e07b" } new_table_name: "default.Table_6a4b8c6d8e4e404981d168241fa84606" modify_external_catalogs: true
I20250115 14:51:57.212538 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0d97fa57484b9d8cb879fedda61612 (DEFAULT_TABLE table=default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.212519 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c6bb6ec57a9240c48a9f5607865e62c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:57.212842 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0d97fa57484b9d8cb879fedda61612. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:57.212982 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6bb6ec57a9240c48a9f5607865e62c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.214026 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7628ff855c7f4396a10a9fb5e7a9acba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.214526 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7628ff855c7f4396a10a9fb5e7a9acba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.217501 749 tablet_service.cc:1467] Processing CreateTablet for tablet 5b85d63974ab418eb1921dd605fbf2b0 (DEFAULT_TABLE table=default.Table_854a113be33b4b698a50ae5bd10a2fff [id=f1ce643dc59040148201fdd0c049fb20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.217834 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b85d63974ab418eb1921dd605fbf2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.224400 744 tablet_service.cc:1467] Processing CreateTablet for tablet e034d7761a5247c79414b7112a1c97b2 (DEFAULT_TABLE table=default.Table_4deac3b7f7124b6aa49514a17811b1c3 [id=efb16fff4d6c4f44af7dfaaa00d4f444]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.227412 740 tablet_service.cc:1467] Processing CreateTablet for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 (DEFAULT_TABLE table=default.Table_b3a6b57353484cdf8f2dfde2738ae0c2 [id=5249bd4dd6f14f9bbfced76f1d336c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.228273 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e034d7761a5247c79414b7112a1c97b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.231642 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.233724 1164 tablet_bootstrap.cc:492] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.234525 1164 ts_tablet_manager.cc:1397] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.181s user 0.009s sys 0.005s
I20250115 14:51:57.235569 1164 raft_consensus.cc:357] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.237783 1164 raft_consensus.cc:383] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.238633 1164 raft_consensus.cc:738] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.239132 1164 consensus_queue.cc:260] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.244837 1164 ts_tablet_manager.cc:1428] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250115 14:51:57.253059 1164 tablet_bootstrap.cc:492] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.206583 1021 tablet_service.cc:1467] Processing CreateTablet for tablet efb8b001dd66456d9e49e9e8a6848ccf (DEFAULT_TABLE table=default.Table_d3380e50077a425fbd4ded35b60ba49c [id=877cb5d7df394006822f9a0e8a133dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.254983 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efb8b001dd66456d9e49e9e8a6848ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.256597 1166 tablet_replica.cc:331] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.256834 1166 raft_consensus.cc:2238] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.256983 1166 raft_consensus.cc:2267] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.257459 1164 tablet_bootstrap.cc:654] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.257519 1166 ts_tablet_manager.cc:1905] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.206930 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 (DEFAULT_TABLE table=default.Table_b3a6b57353484cdf8f2dfde2738ae0c2 [id=5249bd4dd6f14f9bbfced76f1d336c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.259989 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.263075 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 8d43cd66b18f4649894a629637037822 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]) successfully deleted
W20250115 14:51:57.263322 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d97c2f443ab040f9bc1924f45379165e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d97c2f443ab040f9bc1924f45379165e already in progress: creating tablet
W20250115 14:51:57.263513 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 86f0144904ae4008945ccf325039dac1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86f0144904ae4008945ccf325039dac1 already in progress: creating tablet
W20250115 14:51:57.263692 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ab1a38186c5042ff851f57979ab80dbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab1a38186c5042ff851f57979ab80dbc already in progress: creating tablet
W20250115 14:51:57.263877 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1e7580b6245b41628874d9dcbc3366e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e7580b6245b41628874d9dcbc3366e2 already in progress: creating tablet
W20250115 14:51:57.264137 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b2c385edf4414157845b971aaf366831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2c385edf4414157845b971aaf366831 already in progress: creating tablet
I20250115 14:51:57.207221 1011 tablet_service.cc:1467] Processing CreateTablet for tablet e034d7761a5247c79414b7112a1c97b2 (DEFAULT_TABLE table=default.Table_4deac3b7f7124b6aa49514a17811b1c3 [id=efb16fff4d6c4f44af7dfaaa00d4f444]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.264664 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e034d7761a5247c79414b7112a1c97b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.264827 756 tablet_service.cc:1467] Processing CreateTablet for tablet efb8b001dd66456d9e49e9e8a6848ccf (DEFAULT_TABLE table=default.Table_d3380e50077a425fbd4ded35b60ba49c [id=877cb5d7df394006822f9a0e8a133dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.265233 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efb8b001dd66456d9e49e9e8a6848ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.265090 452 catalog_manager.cc:5526] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:57.265517 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4239747374aa4d6d866961430786677c (table default.Table_d810e40dae3d42e38f38ebf32e62305e [id=8941c6c27037475990cf318614854a27]). Expected version 1 got 0
I20250115 14:51:57.265738 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 6b60ff80679a470cb4bdced71cc14195 (table default.Table_c088519ba27c457983f341363f8a109e [id=201c214e2eab4873af2095e83dc88b23]). Expected version 2 got 0
I20250115 14:51:57.266445 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2711a7d437f44b2fa11504d8e400e360
I20250115 14:51:57.207495 1025 tablet_service.cc:1467] Processing CreateTablet for tablet ec2a8000f5474f62aae52c8e88e45276 (DEFAULT_TABLE table=default.Table_01f73e93e1ba462097edfe0e8aaba189 [id=1b0b109197b0474395a710dd740f944f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.267572 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 2711a7d437f44b2fa11504d8e400e360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.208087 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 43605c995df4400e91139199c9514bfd (DEFAULT_TABLE table=default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.208397 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 5b85d63974ab418eb1921dd605fbf2b0 (DEFAULT_TABLE table=default.Table_854a113be33b4b698a50ae5bd10a2fff [id=f1ce643dc59040148201fdd0c049fb20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.267930 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec2a8000f5474f62aae52c8e88e45276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.268188 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b85d63974ab418eb1921dd605fbf2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.207811 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0d97fa57484b9d8cb879fedda61612 (DEFAULT_TABLE table=default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.272447 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43605c995df4400e91139199c9514bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.273003 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0d97fa57484b9d8cb879fedda61612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.227916 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 (DEFAULT_TABLE table=default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.279943 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6cc08065494d9e85eeff5044b44ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.208699 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee8b63cc21145d8993b058d8c6d23c7 (DEFAULT_TABLE table=default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.284780 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee8b63cc21145d8993b058d8c6d23c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.285650 910 raft_consensus.cc:3054] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.291224 913 raft_consensus.cc:2463] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
W20250115 14:51:57.292105 983 consensus_peers.cc:487] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 -> Peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Couldn't send request to peer 73c86158cc2641f182ea50a06634592d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bcfab1e1f0349739eff636fb1b04806. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:57.293433 983 leader_election.cc:304] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:57.294283 1231 raft_consensus.cc:2798] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:57.294369 1231 raft_consensus.cc:491] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:57.294468 1231 raft_consensus.cc:3054] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:57.295557 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f847f92cd122423aa93f1afa701b0bbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f847f92cd122423aa93f1afa701b0bbb already in progress: creating tablet
I20250115 14:51:57.296396 1540 consensus_queue.cc:1035] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:51:57.298943 710 leader_election.cc:343] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: f6205e20a1984ccabccc45b93f9c7fa8
I20250115 14:51:57.298842 1231 raft_consensus.cc:513] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.300068 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeca9975f1945ad956f47db1ea16a73" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:57.300586 984 leader_election.cc:343] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.300662 1162 raft_consensus.cc:357] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.300859 1162 raft_consensus.cc:383] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.300943 1162 raft_consensus.cc:738] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.301092 1231 leader_election.cc:290] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.301169 1162 consensus_queue.cc:260] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.301710 1162 ts_tablet_manager.cc:1428] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.134s user 0.000s sys 0.001s
I20250115 14:51:57.304041 1165 ts_tablet_manager.cc:1939] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.308651 1624 raft_consensus.cc:491] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.312779 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ad45cc2411ad4ee1a65b9db7a0a6e4b5" } new_table_name: "default.Table_c3bef2a77d0640848112f55aa0ed662b" modify_external_catalogs: true
I20250115 14:51:57.313165 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_97c066067f7a49cab2dd5bdade9f6de2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.313714 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97c066067f7a49cab2dd5bdade9f6de2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.313716 740 tablet_service.cc:1514] Processing DeleteTablet for tablet d77ff794a83348c29391786666987db1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.314402 1164 tablet_bootstrap.cc:492] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.314376 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c5c33df1942c4b90a8069108477e91f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:57.314550 1164 ts_tablet_manager.cc:1397] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.062s user 0.020s sys 0.019s
W20250115 14:51:57.314868 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5c33df1942c4b90a8069108477e91f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.314926 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3529669cc14746fda74ad45528cbcf4c" } new_table_name: "default.Table_a8abdc319d2c413ba113c127fbfd5971" modify_external_catalogs: true
I20250115 14:51:57.315351 1164 raft_consensus.cc:357] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.315558 1164 raft_consensus.cc:383] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.315645 1164 raft_consensus.cc:738] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.315865 1164 consensus_queue.cc:260] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.316869 1164 ts_tablet_manager.cc:1428] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:57.318115 1162 tablet_bootstrap.cc:492] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:57.319984 1162 tablet_bootstrap.cc:654] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:57.321514 848 leader_election.cc:343] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.323120 1164 tablet_bootstrap.cc:492] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.327922 1164 tablet_bootstrap.cc:654] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:57.327992 710 leader_election.cc:343] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: f6205e20a1984ccabccc45b93f9c7fa8
I20250115 14:51:57.329394 710 leader_election.cc:304] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:57.329617 710 raft_consensus.cc:2681] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:51:57.330169 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 2711a7d437f44b2fa11504d8e400e360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.331482 1165 tablet_replica.cc:331] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.332655 1165 raft_consensus.cc:2238] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:57.332976 1165 pending_rounds.cc:62] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:57.333071 1165 pending_rounds.cc:69] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558325986177024 op_type: NO_OP noop_request { }
I20250115 14:51:57.333194 1165 raft_consensus.cc:2883] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:57.333317 1165 raft_consensus.cc:2267] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.336526 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 1e46b1c485e84e09a99666934758a798
I20250115 14:51:57.336835 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f60611e6b8324d44b54218a523f2fedd" } new_table_name: "default.Table_7ca2cd26067f4a1997afc7ca98907d59" modify_external_catalogs: true
I20250115 14:51:57.339246 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e3e5361d9cb144999d02ff4df3e1999b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.339771 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3e5361d9cb144999d02ff4df3e1999b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.343261 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b6a520bf69264c44b1da30c0ffb41cc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.343719 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6a520bf69264c44b1da30c0ffb41cc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.349284 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aeca9975f1945ad956f47db1ea16a73" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:57.351711 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_90f2e2c555f84fe39b5b478581ef0e44" } new_table_name: "default.Table_cac9e05c8cae4cab9a55c08b3dc25648" modify_external_catalogs: true
I20250115 14:51:57.351969 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet d86075ae5b894d5986ac4bd538f4d42a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:57.352799 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7233ebf3bae1462d8eeb3669b39944c0 consensus state has no leader
I20250115 14:51:57.353135 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_10deaf92a667422fa029a731f755dfef" } modify_external_catalogs: true
I20250115 14:51:57.353284 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_10deaf92a667422fa029a731f755dfef" } modify_external_catalogs: true
I20250115 14:51:57.355990 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_53f051fa3f6b4051882de44c9baff2aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.356467 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53f051fa3f6b4051882de44c9baff2aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.356549 1628 raft_consensus.cc:491] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.357107 1628 raft_consensus.cc:513] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.357730 1628 leader_election.cc:290] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:57.357954 1165 ts_tablet_manager.cc:1905] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.358960 1162 tablet_bootstrap.cc:492] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:57.359093 1162 ts_tablet_manager.cc:1397] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.005s
I20250115 14:51:57.359824 1162 raft_consensus.cc:357] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.360025 1162 raft_consensus.cc:383] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.360113 1162 raft_consensus.cc:738] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.361582 1624 raft_consensus.cc:513] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:57.361897 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 147049us
W20250115 14:51:57.362967 710 leader_election.cc:343] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d77ff794a83348c29391786666987db1
I20250115 14:51:57.362898 1162 consensus_queue.cc:260] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:57.364921 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be63bae96c834a8bbbb1174c027935a2 consensus state has no leader
I20250115 14:51:57.364981 1166 ts_tablet_manager.cc:1918] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.365038 1162 ts_tablet_manager.cc:1428] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:51:57.365124 1166 log.cc:1198] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f6205e20a1984ccabccc45b93f9c7fa8
I20250115 14:51:57.365330 1162 tablet_bootstrap.cc:492] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:57.365464 1166 ts_tablet_manager.cc:1939] T f6205e20a1984ccabccc45b93f9c7fa8 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.366353 1166 tablet_replica.cc:331] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.366520 1166 raft_consensus.cc:2238] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.366670 1166 raft_consensus.cc:2267] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.367522 756 tablet_service.cc:1514] Processing DeleteTablet for tablet c56fe5b5b59b4017be3cb0ac877479b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.375180 1167 ts_tablet_manager.cc:1918] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:57.367528 1166 ts_tablet_manager.cc:1905] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.382238 910 raft_consensus.cc:2463] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:57.382481 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f6205e20a1984ccabccc45b93f9c7fa8 (table default.Table_c68a808d4c4744fe92a97753d517180a [id=7222964ef7514b2ba94aa1527810947a]) successfully deleted
I20250115 14:51:57.382566 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_950b48d1b1344479a7229041f044bbb6" } modify_external_catalogs: true
I20250115 14:51:57.382752 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_950b48d1b1344479a7229041f044bbb6" } modify_external_catalogs: true
I20250115 14:51:57.384562 1166 ts_tablet_manager.cc:1918] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.384724 1166 log.cc:1198] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ad67354f35ce4564b88dbdffcabf3564
I20250115 14:51:57.384750 983 leader_election.cc:304] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:57.384967 983 raft_consensus.cc:2681] T 5027fba8fed34db59b484918256abc45 P c238c037a1c74725a001e2b4a1ff4304: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:51:57.385126 1166 ts_tablet_manager.cc:1939] T ad67354f35ce4564b88dbdffcabf3564 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.385478 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bba588d115d34226951843abdb9b6ebe" } modify_external_catalogs: true
I20250115 14:51:57.385644 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bba588d115d34226951843abdb9b6ebe" } modify_external_catalogs: true
W20250115 14:51:57.387152 848 leader_election.cc:343] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:57.387195 710 leader_election.cc:343] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: d77ff794a83348c29391786666987db1
I20250115 14:51:57.387271 848 leader_election.cc:304] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:57.387338 710 leader_election.cc:304] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:57.387638 848 leader_election.cc:343] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.387701 1628 raft_consensus.cc:2743] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:57.388198 1594 raft_consensus.cc:2743] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:57.390377 1657 consensus_queue.cc:1035] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250115 14:51:57.392724 1166 tablet_replica.cc:331] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.392824 1167 log.cc:1198] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/259ac764ca844df9b6c0ce1688a188da
I20250115 14:51:57.393520 913 raft_consensus.cc:3054] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.393857 1166 raft_consensus.cc:2238] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.394053 1166 raft_consensus.cc:2267] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:57.394063 847 consensus_peers.cc:487] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4fb2d33f3c745beb4a4ba036e23a8e0. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:57.393589 1164 tablet_bootstrap.cc:492] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.394351 1164 ts_tablet_manager.cc:1397] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.018s
I20250115 14:51:57.395148 1164 raft_consensus.cc:357] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.395409 1164 raft_consensus.cc:383] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.395506 1164 raft_consensus.cc:738] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.395727 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fe77f8263ada4a8390e949b24c6ae559"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:57.395761 1164 consensus_queue.cc:260] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:57.396273 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe77f8263ada4a8390e949b24c6ae559 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.397030 1164 ts_tablet_manager.cc:1428] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:57.397814 1164 tablet_bootstrap.cc:492] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.399431 1164 tablet_bootstrap.cc:654] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.400453 1166 ts_tablet_manager.cc:1905] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.401630 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7309ecb11d844848638c042c9bbe5f1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:57.407017 1166 ts_tablet_manager.cc:1918] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:57.410452 847 leader_election.cc:343] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.410755 1166 log.cc:1198] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d77ff794a83348c29391786666987db1
I20250115 14:51:57.411391 1164 tablet_bootstrap.cc:492] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.411553 1164 ts_tablet_manager.cc:1397] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250115 14:51:57.412940 1164 raft_consensus.cc:357] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.413203 1164 raft_consensus.cc:383] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.413326 1164 raft_consensus.cc:738] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.413611 1164 consensus_queue.cc:260] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.414299 1164 ts_tablet_manager.cc:1428] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250115 14:51:57.414718 1166 ts_tablet_manager.cc:1939] T d77ff794a83348c29391786666987db1 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.415606 1165 ts_tablet_manager.cc:1918] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:57.415002 1164 tablet_bootstrap.cc:492] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.415737 1165 log.cc:1198] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9bcfab1e1f0349739eff636fb1b04806
I20250115 14:51:57.417400 1164 tablet_bootstrap.cc:654] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.418082 1231 raft_consensus.cc:491] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.418118 913 raft_consensus.cc:2463] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:57.418234 1231 raft_consensus.cc:513] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.418313 1168 tablet_bootstrap.cc:492] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:57.418823 1231 leader_election.cc:290] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.419173 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b60ff80679a470cb4bdced71cc14195" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:57.419340 983 leader_election.cc:304] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:57.419754 984 leader_election.cc:343] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.419936 1231 raft_consensus.cc:2798] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:57.420061 1231 raft_consensus.cc:695] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:57.420334 1231 consensus_queue.cc:237] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:57.421423 847 consensus_peers.cc:487] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86f0144904ae4008945ccf325039dac1. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:57.422448 1165 ts_tablet_manager.cc:1939] T 9bcfab1e1f0349739eff636fb1b04806 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.424319 1167 ts_tablet_manager.cc:1939] T 259ac764ca844df9b6c0ce1688a188da P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:57.408921 1652 tablet.cc:1721] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:57.410494 1624 leader_election.cc:290] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:57.428167 1225 raft_consensus.cc:491] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.428328 1225 raft_consensus.cc:513] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.428958 1166 tablet_replica.cc:331] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.428995 1225 leader_election.cc:290] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.429136 1166 raft_consensus.cc:2238] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.429142 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9bcfab1e1f0349739eff636fb1b04806 (table default.Table_fa872457db7140b6a13bfc998e32581b [id=4e1c43b0e14844babf0bb0cf8032a3fa]) successfully deleted
I20250115 14:51:57.429299 1166 raft_consensus.cc:2267] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:57.429975 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f847f92cd122423aa93f1afa701b0bbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f847f92cd122423aa93f1afa701b0bbb already in progress: creating tablet
W20250115 14:51:57.430717 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e7100e6a51ed4e178e86138f10d5ffc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7100e6a51ed4e178e86138f10d5ffc2 already in progress: creating tablet
W20250115 14:51:57.431398 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7d07686c40324233a49e4c49aeed73f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d07686c40324233a49e4c49aeed73f6 already in progress: creating tablet
I20250115 14:51:57.431828 1165 tablet_replica.cc:331] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.432025 1165 raft_consensus.cc:2238] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:57.432228 1165 pending_rounds.cc:62] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:57.432322 1165 pending_rounds.cc:69] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558327915298816 op_type: NO_OP noop_request { }
I20250115 14:51:57.432430 1165 raft_consensus.cc:2883] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:57.432449 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86075ae5b894d5986ac4bd538f4d42a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:57.432544 1165 raft_consensus.cc:2267] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.432991 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b60ff80679a470cb4bdced71cc14195" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:57.433215 913 raft_consensus.cc:2463] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:57.433316 1166 ts_tablet_manager.cc:1905] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:57.434875 984 leader_election.cc:343] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.427912 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 287) took 1014 ms. Trace:
I20250115 14:51:57.436482 983 leader_election.cc:304] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:57.436909 1225 raft_consensus.cc:2798] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:57.437021 1225 raft_consensus.cc:491] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:57.437106 1225 raft_consensus.cc:3054] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.438500 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ad67354f35ce4564b88dbdffcabf3564 (table default.Table_bbf01a6c24b9449b8da3ee9428b7bd98 [id=23302b50545244ddb17c074740aacccd]) successfully deleted
W20250115 14:51:57.438717 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7d07686c40324233a49e4c49aeed73f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d07686c40324233a49e4c49aeed73f6 already in progress: creating tablet
W20250115 14:51:57.438913 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 37ad2b7350294375bd30fb4b46906491 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37ad2b7350294375bd30fb4b46906491 already in progress: creating tablet
W20250115 14:51:57.439076 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 629873c00550408798b00489ece4ea5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 629873c00550408798b00489ece4ea5f already in progress: creating tablet
W20250115 14:51:57.439239 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e7100e6a51ed4e178e86138f10d5ffc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7100e6a51ed4e178e86138f10d5ffc2 already in progress: creating tablet
W20250115 14:51:57.439407 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 36685a04e7454f6592f6ab83c356f466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36685a04e7454f6592f6ab83c356f466 already in progress: creating tablet
I20250115 14:51:57.439592 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d77ff794a83348c29391786666987db1 (table default.Table_26cfaab25925425bb14d5384a6ce1827 [id=6c9d98b8d6de494a9c6211022d1fc946]) successfully deleted
I20250115 14:51:57.439535 1225 raft_consensus.cc:513] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.439622 1166 ts_tablet_manager.cc:1918] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.439730 1166 log.cc:1198] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/c56fe5b5b59b4017be3cb0ac877479b4
W20250115 14:51:57.439791 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2711a7d437f44b2fa11504d8e400e360 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2711a7d437f44b2fa11504d8e400e360 already in progress: creating tablet
I20250115 14:51:57.440068 1166 ts_tablet_manager.cc:1939] T c56fe5b5b59b4017be3cb0ac877479b4 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.440752 1165 ts_tablet_manager.cc:1905] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.440759 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b60ff80679a470cb4bdced71cc14195" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:57.441027 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 consensus state has no leader
W20250115 14:51:57.441248 984 leader_election.cc:343] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.441552 1225 leader_election.cc:290] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.444022 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3ef67fd49f0b4888a1f78b8e3b4558f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.444653 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ef67fd49f0b4888a1f78b8e3b4558f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.449512 1168 ts_tablet_manager.cc:1397] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.289s user 0.007s sys 0.015s
I20250115 14:51:57.451061 1165 ts_tablet_manager.cc:1918] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:57.457634 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d3b4117171441a987ac043931fc6897 with delete_type TABLET_DATA_DELETED (replaced by tablet 906fd4d0aed1497ebe0c7316aa6424bb) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.457938 1166 tablet_replica.cc:331] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.458189 1166 raft_consensus.cc:2238] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.458482 1166 raft_consensus.cc:2267] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.459385 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet c56fe5b5b59b4017be3cb0ac877479b4 (table default.Table_740738b002f34e2dba618cd146ff8dcd [id=3e278eb470174b78a8cdaab8d9ab2464]) successfully deleted
I20250115 14:51:57.460217 1166 ts_tablet_manager.cc:1905] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:57.463614 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5ffa997ca6c46559830b1dedf2d1a66 consensus state has no leader
I20250115 14:51:57.465140 1164 tablet_bootstrap.cc:492] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.465332 1164 ts_tablet_manager.cc:1397] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.007s
I20250115 14:51:57.463582 1167 rpcz_store.cc:276] 0115 14:51:56.417049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:56.417091 (+ 42us) service_pool.cc:225] Handling call
0115 14:51:56.872948 (+455857us) ts_tablet_manager.cc:1145] Deleting tablet 259ac764ca844df9b6c0ce1688a188da
0115 14:51:56.872955 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:57.037279 (+164324us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:57.313102 (+275823us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:57.424592 (+111490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":455675}
I20250115 14:51:57.466243 1164 raft_consensus.cc:357] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.466557 1164 raft_consensus.cc:383] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.466673 1164 raft_consensus.cc:738] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.466998 1164 consensus_queue.cc:260] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.467613 1164 ts_tablet_manager.cc:1428] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:57.467932 1164 tablet_bootstrap.cc:492] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.468349 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 69aed449a776493b8bdc7a31b47b62e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.460121 1162 tablet_bootstrap.cc:654] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.471901 1164 tablet_bootstrap.cc:654] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.472095 1165 log.cc:1198] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/36685a04e7454f6592f6ab83c356f466
I20250115 14:51:57.472781 1165 ts_tablet_manager.cc:1939] T 36685a04e7454f6592f6ab83c356f466 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.473863 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 36685a04e7454f6592f6ab83c356f466 (table default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]) successfully deleted
I20250115 14:51:57.474063 1165 tablet_replica.cc:331] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.474269 1165 raft_consensus.cc:2238] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.474413 1165 raft_consensus.cc:2267] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:57.474486 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 629873c00550408798b00489ece4ea5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 629873c00550408798b00489ece4ea5f already in progress: creating tablet
W20250115 14:51:57.474720 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 37ad2b7350294375bd30fb4b46906491 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37ad2b7350294375bd30fb4b46906491 already in progress: creating tablet
W20250115 14:51:57.474929 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ab1a38186c5042ff851f57979ab80dbc because the tablet was not found. No further retry: Not found: Tablet not found: ab1a38186c5042ff851f57979ab80dbc
I20250115 14:51:57.477595 1165 ts_tablet_manager.cc:1905] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.479843 1166 ts_tablet_manager.cc:1918] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.480607 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b60ff80679a470cb4bdced71cc14195" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:57.480815 910 raft_consensus.cc:3054] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.484553 1167 tablet_replica.cc:331] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:57.484980 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86075ae5b894d5986ac4bd538f4d42a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:57.485330 1167 raft_consensus.cc:2238] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.485848 1167 raft_consensus.cc:2267] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.487596 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:51:57.480378 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 24751e606ad94675b650d67a3738481e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.488591 1166 log.cc:1198] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9d3b4117171441a987ac043931fc6897
I20250115 14:51:57.489010 1166 ts_tablet_manager.cc:1939] T 9d3b4117171441a987ac043931fc6897 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.490921 1166 tablet_replica.cc:331] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.491359 1166 raft_consensus.cc:2238] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.491788 1166 raft_consensus.cc:2267] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.492550 913 raft_consensus.cc:2463] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:57.492733 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]). Expected version 1 got 0
I20250115 14:51:57.492973 983 leader_election.cc:304] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:57.493332 447 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d7309ecb11d844848638c042c9bbe5f1 consensus state has no leader
I20250115 14:51:57.493444 1225 raft_consensus.cc:2798] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:57.493531 1225 raft_consensus.cc:491] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:57.494171 1225 raft_consensus.cc:3054] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.494451 1165 ts_tablet_manager.cc:1918] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.494576 1165 log.cc:1198] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e50462626f914fb1b0f440c48f30e72d
I20250115 14:51:57.494593 1168 raft_consensus.cc:357] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.494796 1168 raft_consensus.cc:383] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.494872 1168 raft_consensus.cc:738] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.494947 1165 ts_tablet_manager.cc:1939] T e50462626f914fb1b0f440c48f30e72d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.495080 1168 consensus_queue.cc:260] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.495548 1168 ts_tablet_manager.cc:1428] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20250115 14:51:57.495800 1168 tablet_bootstrap.cc:492] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:57.495903 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e50462626f914fb1b0f440c48f30e72d (table default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]) successfully deleted
I20250115 14:51:57.496623 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9e95d09efd2e44cc81cdcbf87ee453fe (DEFAULT_TABLE table=default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.496992 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e95d09efd2e44cc81cdcbf87ee453fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.497365 1168 tablet_bootstrap.cc:654] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.499446 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 4341977f363d40eeac71b7d33d1b43ac (DEFAULT_TABLE table=default.Table_d70c898088424b64902d7ba0be41017d [id=b1e6efb09adf4c898362ed62f9fb98f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.499863 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4341977f363d40eeac71b7d33d1b43ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.500561 1011 tablet_service.cc:1467] Processing CreateTablet for tablet be2fd31f74a94787a5e8ccae3c5ef245 (DEFAULT_TABLE table=default.Table_fcacad621d8e46eebb110daa55789690 [id=724c015c8f3f4aa581ac6faa84187b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.500907 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be2fd31f74a94787a5e8ccae3c5ef245. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.502849 756 tablet_service.cc:1467] Processing CreateTablet for tablet 9e95d09efd2e44cc81cdcbf87ee453fe (DEFAULT_TABLE table=default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.503214 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e95d09efd2e44cc81cdcbf87ee453fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.502429 1167 ts_tablet_manager.cc:1905] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.504515 1021 tablet_service.cc:1467] Processing CreateTablet for tablet fa57c1fa96594a40b2551ad503127e4f (DEFAULT_TABLE table=default.Table_00dac00a165544449ce9b8dff8efab98 [id=8242d7a08fcd4d4e836f6b70b1bacdc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.504861 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa57c1fa96594a40b2551ad503127e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.505000 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 84844b5f50af408ab7dda73f7421cd9d (DEFAULT_TABLE table=default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.505323 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84844b5f50af408ab7dda73f7421cd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.508540 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a600fa0f0ec049dca2803bf7d2c74a95 (DEFAULT_TABLE table=default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d [id=ab53ab1ed1b34dafa524ccb35291fd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.508934 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a600fa0f0ec049dca2803bf7d2c74a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.510658 756 tablet_service.cc:1467] Processing CreateTablet for tablet be2fd31f74a94787a5e8ccae3c5ef245 (DEFAULT_TABLE table=default.Table_fcacad621d8e46eebb110daa55789690 [id=724c015c8f3f4aa581ac6faa84187b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.511188 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be2fd31f74a94787a5e8ccae3c5ef245. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.512735 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 90c1602aef514dd29d5d2cffb043dbb6 (DEFAULT_TABLE table=default.Table_950b48d1b1344479a7229041f044bbb6 [id=65eabb2be7d1479f889da99e50da551f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.513038 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c1602aef514dd29d5d2cffb043dbb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.514272 1166 ts_tablet_manager.cc:1905] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.516592 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 282d988bc41443c381bb0ba69b618f5d (DEFAULT_TABLE table=default.Table_05f796ba50b14c1f973f89c85c133164 [id=83320921ba86467b9ad31fde698cb2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.516935 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 282d988bc41443c381bb0ba69b618f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.516880 740 tablet_service.cc:1467] Processing CreateTablet for tablet fa57c1fa96594a40b2551ad503127e4f (DEFAULT_TABLE table=default.Table_00dac00a165544449ce9b8dff8efab98 [id=8242d7a08fcd4d4e836f6b70b1bacdc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.517230 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa57c1fa96594a40b2551ad503127e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.519207 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9d3b4117171441a987ac043931fc6897 (table default.Table_8d37d2aba1094de3a9bc2eabee94643d [id=207a93a004da4fbc91db802c6939059a]) successfully deleted
I20250115 14:51:57.520395 910 raft_consensus.cc:2463] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:57.520531 1012 tablet_service.cc:1467] Processing CreateTablet for tablet e5a059e2f0654a4aafe9e8082b8fde7a (DEFAULT_TABLE table=default.Table_874111e557c1400bb8fb21c733becad1 [id=54db87fdb55743b8a485808e00abdbee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.520586 739 tablet_service.cc:1467] Processing CreateTablet for tablet a600fa0f0ec049dca2803bf7d2c74a95 (DEFAULT_TABLE table=default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d [id=ab53ab1ed1b34dafa524ccb35291fd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.520862 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a059e2f0654a4aafe9e8082b8fde7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.520910 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a600fa0f0ec049dca2803bf7d2c74a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.524866 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 629873c00550408798b00489ece4ea5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.525291 892 tablet_service.cc:1514] Processing DeleteTablet for tablet f50edca843e54f4b8e803d4d180b736d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.525803 892 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 (DEFAULT_TABLE table=default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.526162 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e6cc08065494d9e85eeff5044b44ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.524269 744 tablet_service.cc:1467] Processing CreateTablet for tablet 4341977f363d40eeac71b7d33d1b43ac (DEFAULT_TABLE table=default.Table_d70c898088424b64902d7ba0be41017d [id=b1e6efb09adf4c898362ed62f9fb98f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.524552 750 tablet_service.cc:1467] Processing CreateTablet for tablet 90c1602aef514dd29d5d2cffb043dbb6 (DEFAULT_TABLE table=default.Table_950b48d1b1344479a7229041f044bbb6 [id=65eabb2be7d1479f889da99e50da551f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.524741 741 tablet_service.cc:1467] Processing CreateTablet for tablet 282d988bc41443c381bb0ba69b618f5d (DEFAULT_TABLE table=default.Table_05f796ba50b14c1f973f89c85c133164 [id=83320921ba86467b9ad31fde698cb2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.526504 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4341977f363d40eeac71b7d33d1b43ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.524914 748 tablet_service.cc:1467] Processing CreateTablet for tablet e5a059e2f0654a4aafe9e8082b8fde7a (DEFAULT_TABLE table=default.Table_874111e557c1400bb8fb21c733becad1 [id=54db87fdb55743b8a485808e00abdbee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.526568 1167 ts_tablet_manager.cc:1918] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.526717 1167 log.cc:1198] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/5027fba8fed34db59b484918256abc45
I20250115 14:51:57.525081 746 tablet_service.cc:1467] Processing CreateTablet for tablet 84844b5f50af408ab7dda73f7421cd9d (DEFAULT_TABLE table=default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.525261 752 tablet_service.cc:1467] Processing CreateTablet for tablet 07a000ab73dd46ac8e134784a156040a (DEFAULT_TABLE table=default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.527422 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 282d988bc41443c381bb0ba69b618f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.529368 983 leader_election.cc:304] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:57.529657 1540 raft_consensus.cc:2798] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:57.531169 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ad2b7350294375bd30fb4b46906491 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.531641 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a059e2f0654a4aafe9e8082b8fde7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.532570 892 tablet_service.cc:1467] Processing CreateTablet for tablet efb8b001dd66456d9e49e9e8a6848ccf (DEFAULT_TABLE table=default.Table_d3380e50077a425fbd4ded35b60ba49c [id=877cb5d7df394006822f9a0e8a133dcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.532931 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efb8b001dd66456d9e49e9e8a6848ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.533280 879 tablet_service.cc:1467] Processing CreateTablet for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 (DEFAULT_TABLE table=default.Table_b3a6b57353484cdf8f2dfde2738ae0c2 [id=5249bd4dd6f14f9bbfced76f1d336c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.533635 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.536525 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84844b5f50af408ab7dda73f7421cd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.538795 883 tablet_service.cc:1467] Processing CreateTablet for tablet e034d7761a5247c79414b7112a1c97b2 (DEFAULT_TABLE table=default.Table_4deac3b7f7124b6aa49514a17811b1c3 [id=efb16fff4d6c4f44af7dfaaa00d4f444]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.539386 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e034d7761a5247c79414b7112a1c97b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.540472 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a000ab73dd46ac8e134784a156040a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.540828 1225 raft_consensus.cc:513] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.541824 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86075ae5b894d5986ac4bd538f4d42a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:57.542349 984 leader_election.cc:343] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.542752 1225 leader_election.cc:290] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.543422 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d86075ae5b894d5986ac4bd538f4d42a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:57.543632 910 raft_consensus.cc:3054] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.545502 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c1602aef514dd29d5d2cffb043dbb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.548995 910 raft_consensus.cc:2463] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:57.549450 983 leader_election.cc:304] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:57.549918 883 tablet_service.cc:1467] Processing CreateTablet for tablet ec2a8000f5474f62aae52c8e88e45276 (DEFAULT_TABLE table=default.Table_01f73e93e1ba462097edfe0e8aaba189 [id=1b0b109197b0474395a710dd740f944f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.550287 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec2a8000f5474f62aae52c8e88e45276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.551519 886 tablet_service.cc:1467] Processing CreateTablet for tablet 3a0d97fa57484b9d8cb879fedda61612 (DEFAULT_TABLE table=default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.551880 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0d97fa57484b9d8cb879fedda61612. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.557539 1167 ts_tablet_manager.cc:1939] T 5027fba8fed34db59b484918256abc45 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:57.557897 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 288) took 1140 ms. Trace:
I20250115 14:51:57.558033 1167 rpcz_store.cc:276] 0115 14:51:56.417792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:56.417832 (+ 40us) service_pool.cc:225] Handling call
0115 14:51:57.465641 (+1047809us) ts_tablet_manager.cc:1145] Deleting tablet 5027fba8fed34db59b484918256abc45
0115 14:51:57.465650 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:57.523552 (+ 57902us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:57.526473 (+ 2921us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:57.557841 (+ 31368us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047615}
I20250115 14:51:57.558434 886 tablet_service.cc:1467] Processing CreateTablet for tablet 43605c995df4400e91139199c9514bfd (DEFAULT_TABLE table=default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.558827 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43605c995df4400e91139199c9514bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.563057 1225 raft_consensus.cc:2798] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:57.563278 1225 raft_consensus.cc:695] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:57.563596 1225 consensus_queue.cc:237] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.564374 1540 raft_consensus.cc:695] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:57.564710 1540 consensus_queue.cc:237] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.568748 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 2711a7d437f44b2fa11504d8e400e360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.568845 890 tablet_service.cc:1467] Processing CreateTablet for tablet 4ee8b63cc21145d8993b058d8c6d23c7 (DEFAULT_TABLE table=default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.569234 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ee8b63cc21145d8993b058d8c6d23c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.573563 1165 tablet_replica.cc:331] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.574048 1165 raft_consensus.cc:2238] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:57.574565 1165 pending_rounds.cc:62] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:57.576445 877 tablet_service.cc:1467] Processing CreateTablet for tablet 9e95d09efd2e44cc81cdcbf87ee453fe (DEFAULT_TABLE table=default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.576892 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e95d09efd2e44cc81cdcbf87ee453fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.576875 1165 pending_rounds.cc:69] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558327418511360 op_type: NO_OP noop_request { }
I20250115 14:51:57.577278 1165 raft_consensus.cc:2883] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:57.577710 1165 raft_consensus.cc:2267] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.578856 1165 ts_tablet_manager.cc:1905] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.584033 1166 ts_tablet_manager.cc:1918] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.585143 1167 tablet_replica.cc:331] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:57.585378 1167 raft_consensus.cc:2238] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:57.585738 1167 pending_rounds.cc:62] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:57.585808 1167 pending_rounds.cc:69] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558326888017920 op_type: NO_OP noop_request { }
I20250115 14:51:57.585960 1167 raft_consensus.cc:2883] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:57.586056 1167 raft_consensus.cc:2267] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.590183 877 tablet_service.cc:1467] Processing CreateTablet for tablet be2fd31f74a94787a5e8ccae3c5ef245 (DEFAULT_TABLE table=default.Table_fcacad621d8e46eebb110daa55789690 [id=724c015c8f3f4aa581ac6faa84187b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.590570 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be2fd31f74a94787a5e8ccae3c5ef245. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.591143 1166 log.cc:1198] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/69aed449a776493b8bdc7a31b47b62e8
I20250115 14:51:57.590178 890 tablet_service.cc:1467] Processing CreateTablet for tablet fa57c1fa96594a40b2551ad503127e4f (DEFAULT_TABLE table=default.Table_00dac00a165544449ce9b8dff8efab98 [id=8242d7a08fcd4d4e836f6b70b1bacdc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.595153 1165 ts_tablet_manager.cc:1918] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:57.595681 1166 ts_tablet_manager.cc:1939] T 69aed449a776493b8bdc7a31b47b62e8 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.597821 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4abde784f9c948cb96e6cc93c30817bd
I20250115 14:51:57.598800 1166 tablet_replica.cc:331] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
W20250115 14:51:57.598948 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:40227, user_credentials={real_user=slave}} blocked reactor thread for 78877.7us
I20250115 14:51:57.599557 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abde784f9c948cb96e6cc93c30817bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.599658 1166 raft_consensus.cc:2238] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.600445 1166 raft_consensus.cc:2267] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.601460 1166 ts_tablet_manager.cc:1905] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.604583 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4821224ac8f1472d8b1e8afc06a118a3 (table default.Table_721928c380b840ad84f9f5030b52632f [id=c5f831177088497dbf1fee130e231e37]). Expected version 1 got 0
I20250115 14:51:57.597847 1653 tablet.cc:1721] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:57.598788 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abde784f9c948cb96e6cc93c30817bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.611577 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 69aed449a776493b8bdc7a31b47b62e8 (table default.Table_68b340d318b14c078d634b22c471d979 [id=e2d228ef5e814fd9bf50e2ed7e89be4d]) successfully deleted
I20250115 14:51:57.613065 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 07a000ab73dd46ac8e134784a156040a (DEFAULT_TABLE table=default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.613229 1165 log.cc:1198] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b25dce523e724066b203e6111cad8111
I20250115 14:51:57.613464 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a000ab73dd46ac8e134784a156040a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.613713 1165 ts_tablet_manager.cc:1939] T b25dce523e724066b203e6111cad8111 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.617502 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b25dce523e724066b203e6111cad8111 (table default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]) successfully deleted
I20250115 14:51:57.618558 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:51:57.618782 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4b90205ef15f470a99c013e7f2cf99c6 (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]). Expected version 3 got 0
I20250115 14:51:57.618949 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 83415f1c07ad495cabd287bdcd54dad4 (DEFAULT_TABLE table=default.Table_458a573b4a4b4714950723b426f0d265 [id=1fd54f7dafdc4777bd06982391452889]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.619109 1166 ts_tablet_manager.cc:1918] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.619236 1166 log.cc:1198] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/24751e606ad94675b650d67a3738481e
I20250115 14:51:57.619294 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83415f1c07ad495cabd287bdcd54dad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.619535 1166 ts_tablet_manager.cc:1939] T 24751e606ad94675b650d67a3738481e P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.621642 1164 tablet_bootstrap.cc:492] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.621788 1164 ts_tablet_manager.cc:1397] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.154s user 0.012s sys 0.004s
I20250115 14:51:57.622143 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9a090445edb041279a25064152cecee2" } modify_external_catalogs: true
I20250115 14:51:57.622313 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9a090445edb041279a25064152cecee2" } modify_external_catalogs: true
I20250115 14:51:57.622704 1164 raft_consensus.cc:357] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.622888 1164 raft_consensus.cc:383] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.622962 1164 raft_consensus.cc:738] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.622992 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b2a11b7b0dd6494daf7b017edd4317a5 (DEFAULT_TABLE table=default.Table_8e068b869e304185abaccf2d63685cd3 [id=5a50430da9af480282aa31475f196737]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.623186 1164 consensus_queue.cc:260] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.623318 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a11b7b0dd6494daf7b017edd4317a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.625177 1164 ts_tablet_manager.cc:1428] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:57.625094 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_66c26e1f10144558b65efccffc30c61f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:57.625427 1164 tablet_bootstrap.cc:492] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:51:57.625591 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66c26e1f10144558b65efccffc30c61f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.627389 1162 tablet_bootstrap.cc:492] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:57.627548 1162 ts_tablet_manager.cc:1397] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.262s user 0.002s sys 0.005s
I20250115 14:51:57.627664 1165 tablet_replica.cc:331] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.627846 1165 raft_consensus.cc:2238] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:57.628072 1165 pending_rounds.cc:62] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:57.628139 1165 pending_rounds.cc:69] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558330555785216 op_type: NO_OP noop_request { }
I20250115 14:51:57.628132 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a43e4255e5c3414aa47bff8bbe111d13" } modify_external_catalogs: true
I20250115 14:51:57.628265 1165 raft_consensus.cc:2883] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:57.628319 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a43e4255e5c3414aa47bff8bbe111d13" } modify_external_catalogs: true
I20250115 14:51:57.628413 1165 raft_consensus.cc:2267] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.628589 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 377d711f052545ddbe499afe7ba3de28 (DEFAULT_TABLE table=default.Table_7ec0e7f2e4de4b14bd6cea8489658118 [id=898cecdad4254997a5bed59d46f7acec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.628965 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 377d711f052545ddbe499afe7ba3de28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.629113 1164 tablet_bootstrap.cc:654] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.630642 887 tablet_service.cc:1467] Processing CreateTablet for tablet 90c1602aef514dd29d5d2cffb043dbb6 (DEFAULT_TABLE table=default.Table_950b48d1b1344479a7229041f044bbb6 [id=65eabb2be7d1479f889da99e50da551f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.628309 1162 raft_consensus.cc:357] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.630930 1162 raft_consensus.cc:383] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.631021 1162 raft_consensus.cc:738] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.631350 885 tablet_service.cc:1467] Processing CreateTablet for tablet e5a059e2f0654a4aafe9e8082b8fde7a (DEFAULT_TABLE table=default.Table_874111e557c1400bb8fb21c733becad1 [id=54db87fdb55743b8a485808e00abdbee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.631759 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4341977f363d40eeac71b7d33d1b43ac (DEFAULT_TABLE table=default.Table_d70c898088424b64902d7ba0be41017d [id=b1e6efb09adf4c898362ed62f9fb98f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.632031 876 tablet_service.cc:1467] Processing CreateTablet for tablet a600fa0f0ec049dca2803bf7d2c74a95 (DEFAULT_TABLE table=default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d [id=ab53ab1ed1b34dafa524ccb35291fd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.632978 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4dff2e869f8141e09dd757d6e47118f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.633458 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4dff2e869f8141e09dd757d6e47118f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.635869 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_040aa3c43ac34524823fe14b90fe4b9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.636344 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_040aa3c43ac34524823fe14b90fe4b9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.636538 1162 consensus_queue.cc:260] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.637130 1162 ts_tablet_manager.cc:1428] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20250115 14:51:57.637396 1162 tablet_bootstrap.cc:492] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:57.638407 888 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ce445d334842c8a6bacad9f693feda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.638574 878 tablet_service.cc:1467] Processing CreateTablet for tablet 07a000ab73dd46ac8e134784a156040a (DEFAULT_TABLE table=default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.632051 1018 tablet_service.cc:1467] Processing CreateTablet for tablet f06544890cbf460cbe218690e998eb1d (DEFAULT_TABLE table=default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.632411 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 6942d102069d4c33b23fa2c0b29f94a6 (DEFAULT_TABLE table=default.Table_600b928b6de9443db6b3c73398af80b7 [id=fa886a7e8fcd43a096892c6435e13c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.632756 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 92aeca346a7746468dbbff053d933472 (DEFAULT_TABLE table=default.Table_db31d0a74f474c2c8d12130b143b1371 [id=60ff7f730d394990a0263e322d28d7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.633042 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 95718d39a7824517a36c49c3b86fef30 (DEFAULT_TABLE table=default.Table_5d8548b30284461086fdcd351454ad57 [id=d114031263ea4c87a6c4d12c9d6a0d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.633159 1165 ts_tablet_manager.cc:1905] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.633338 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 2484885df0da46f381d625a26e211c1e (DEFAULT_TABLE table=default.Table_2a410c48ef764ba2ac337b8471f11ba1 [id=c29c4105b83d4df9b617997c0c9c5b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.642004 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_23f12c7168a84d5390b6d4ad9ed07ea1" } new_table_name: "default.Table_06a952903c7b4dcc8088b8c9ff5a2ac5" modify_external_catalogs: true
I20250115 14:51:57.642424 1164 tablet_bootstrap.cc:492] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.642701 1164 ts_tablet_manager.cc:1397] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250115 14:51:57.643524 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6942d102069d4c33b23fa2c0b29f94a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.644038 1164 raft_consensus.cc:357] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.644505 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f06544890cbf460cbe218690e998eb1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.644795 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2484885df0da46f381d625a26e211c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.645237 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95718d39a7824517a36c49c3b86fef30. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:57.645565 984 consensus_peers.cc:487] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:57.645632 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92aeca346a7746468dbbff053d933472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.644408 1164 raft_consensus.cc:383] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:51:57.646570 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 1e7580b6245b41628874d9dcbc3366e2 because the tablet was not found. No further retry: Not found: Tablet not found: 1e7580b6245b41628874d9dcbc3366e2
I20250115 14:51:57.646270 1164 raft_consensus.cc:738] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.647486 1164 consensus_queue.cc:260] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.649410 1164 ts_tablet_manager.cc:1428] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250115 14:51:57.651980 1164 tablet_bootstrap.cc:492] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.653344 1162 tablet_bootstrap.cc:654] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.658980 1164 tablet_bootstrap.cc:654] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.662336 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa57c1fa96594a40b2551ad503127e4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.666106 1167 ts_tablet_manager.cc:1905] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.666210 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c1602aef514dd29d5d2cffb043dbb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.666426 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4341977f363d40eeac71b7d33d1b43ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.666520 880 tablet_service.cc:1467] Processing CreateTablet for tablet 83415f1c07ad495cabd287bdcd54dad4 (DEFAULT_TABLE table=default.Table_458a573b4a4b4714950723b426f0d265 [id=1fd54f7dafdc4777bd06982391452889]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.666786 1168 tablet_bootstrap.cc:492] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:57.667904 886 tablet_service.cc:1467] Processing CreateTablet for tablet 5b85d63974ab418eb1921dd605fbf2b0 (DEFAULT_TABLE table=default.Table_854a113be33b4b698a50ae5bd10a2fff [id=f1ce643dc59040148201fdd0c049fb20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.673240 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abde784f9c948cb96e6cc93c30817bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.675268 1164 tablet_bootstrap.cc:492] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.675415 1164 ts_tablet_manager.cc:1397] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.005s
I20250115 14:51:57.676127 884 tablet_service.cc:1467] Processing CreateTablet for tablet 282d988bc41443c381bb0ba69b618f5d (DEFAULT_TABLE table=default.Table_05f796ba50b14c1f973f89c85c133164 [id=83320921ba86467b9ad31fde698cb2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.676132 1164 raft_consensus.cc:357] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.676395 1164 raft_consensus.cc:383] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.676218 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a600fa0f0ec049dca2803bf7d2c74a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.676489 1164 raft_consensus.cc:738] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.676718 1164 consensus_queue.cc:260] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.677132 446 master_service.cc:946] ReplaceTablet: received request to replace tablet f8c81f47cd6e4a308cd62124db38d42f from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:57.678253 1164 ts_tablet_manager.cc:1428] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:57.678416 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7ca2cd26067f4a1997afc7ca98907d59" } new_table_name: "default.Table_092faa925c9f45838cbc3a14b0fc60dd" modify_external_catalogs: true
I20250115 14:51:57.678508 1164 tablet_bootstrap.cc:492] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.679986 1164 tablet_bootstrap.cc:654] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.682376 891 tablet_service.cc:1467] Processing CreateTablet for tablet 84844b5f50af408ab7dda73f7421cd9d (DEFAULT_TABLE table=default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.682433 1168 ts_tablet_manager.cc:1397] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.171s user 0.006s sys 0.006s
I20250115 14:51:57.687036 756 tablet_service.cc:1467] Processing CreateTablet for tablet 83415f1c07ad495cabd287bdcd54dad4 (DEFAULT_TABLE table=default.Table_458a573b4a4b4714950723b426f0d265 [id=1fd54f7dafdc4777bd06982391452889]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.687445 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83415f1c07ad495cabd287bdcd54dad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.694339 744 tablet_service.cc:1467] Processing CreateTablet for tablet b2a11b7b0dd6494daf7b017edd4317a5 (DEFAULT_TABLE table=default.Table_8e068b869e304185abaccf2d63685cd3 [id=5a50430da9af480282aa31475f196737]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.694713 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a11b7b0dd6494daf7b017edd4317a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.698491 879 tablet_service.cc:1467] Processing CreateTablet for tablet 377d711f052545ddbe499afe7ba3de28 (DEFAULT_TABLE table=default.Table_7ec0e7f2e4de4b14bd6cea8489658118 [id=898cecdad4254997a5bed59d46f7acec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.701545 1162 tablet_bootstrap.cc:492] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:57.701705 1162 ts_tablet_manager.cc:1397] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.003s
I20250115 14:51:57.701999 744 tablet_service.cc:1467] Processing CreateTablet for tablet 377d711f052545ddbe499afe7ba3de28 (DEFAULT_TABLE table=default.Table_7ec0e7f2e4de4b14bd6cea8489658118 [id=898cecdad4254997a5bed59d46f7acec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.702450 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 377d711f052545ddbe499afe7ba3de28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.702528 1162 raft_consensus.cc:357] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.702752 1162 raft_consensus.cc:383] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.702845 1162 raft_consensus.cc:738] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.703649 1162 consensus_queue.cc:260] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.704243 1162 ts_tablet_manager.cc:1428] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250115 14:51:57.704391 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b85d63974ab418eb1921dd605fbf2b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.705612 1164 tablet_bootstrap.cc:492] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.706219 1164 ts_tablet_manager.cc:1397] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.012s
I20250115 14:51:57.710915 739 tablet_service.cc:1467] Processing CreateTablet for tablet f06544890cbf460cbe218690e998eb1d (DEFAULT_TABLE table=default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.711249 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f06544890cbf460cbe218690e998eb1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.711310 748 tablet_service.cc:1467] Processing CreateTablet for tablet 6942d102069d4c33b23fa2c0b29f94a6 (DEFAULT_TABLE table=default.Table_600b928b6de9443db6b3c73398af80b7 [id=fa886a7e8fcd43a096892c6435e13c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.711661 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6942d102069d4c33b23fa2c0b29f94a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.712064 1162 tablet_bootstrap.cc:492] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:57.719439 772 raft_consensus.cc:3054] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:57.723621 1162 tablet_bootstrap.cc:654] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.718050 1165 ts_tablet_manager.cc:1918] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:57.724491 1165 log.cc:1198] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9aeca9975f1945ad956f47db1ea16a73
I20250115 14:51:57.725147 1165 ts_tablet_manager.cc:1939] T 9aeca9975f1945ad956f47db1ea16a73 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.729447 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9aeca9975f1945ad956f47db1ea16a73 (table default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666 [id=914642e9e7974554a76a94e205db108f]) successfully deleted
I20250115 14:51:57.730444 1165 tablet_replica.cc:331] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.730895 1165 raft_consensus.cc:2238] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.731384 1165 raft_consensus.cc:2267] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.732161 1165 ts_tablet_manager.cc:1905] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.734970 1164 raft_consensus.cc:357] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.736897 1164 raft_consensus.cc:383] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.737702 1164 raft_consensus.cc:738] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.738811 748 tablet_service.cc:1467] Processing CreateTablet for tablet 92aeca346a7746468dbbff053d933472 (DEFAULT_TABLE table=default.Table_db31d0a74f474c2c8d12130b143b1371 [id=60ff7f730d394990a0263e322d28d7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.739221 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92aeca346a7746468dbbff053d933472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.739714 1164 consensus_queue.cc:260] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.742092 1162 tablet_bootstrap.cc:492] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:57.742293 1162 ts_tablet_manager.cc:1397] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.003s
I20250115 14:51:57.743075 1162 raft_consensus.cc:357] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.743332 1162 raft_consensus.cc:383] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.743449 1162 raft_consensus.cc:738] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.743762 1162 consensus_queue.cc:260] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.744807 1164 ts_tablet_manager.cc:1428] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.038s user 0.002s sys 0.001s
I20250115 14:51:57.747226 1164 tablet_bootstrap.cc:492] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.748864 1162 ts_tablet_manager.cc:1428] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:51:57.749670 744 tablet_service.cc:1467] Processing CreateTablet for tablet 95718d39a7824517a36c49c3b86fef30 (DEFAULT_TABLE table=default.Table_5d8548b30284461086fdcd351454ad57 [id=d114031263ea4c87a6c4d12c9d6a0d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.749765 889 tablet_service.cc:1467] Processing CreateTablet for tablet 6942d102069d4c33b23fa2c0b29f94a6 (DEFAULT_TABLE table=default.Table_600b928b6de9443db6b3c73398af80b7 [id=fa886a7e8fcd43a096892c6435e13c29]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.750106 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95718d39a7824517a36c49c3b86fef30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.750895 892 tablet_service.cc:1467] Processing CreateTablet for tablet b2a11b7b0dd6494daf7b017edd4317a5 (DEFAULT_TABLE table=default.Table_8e068b869e304185abaccf2d63685cd3 [id=5a50430da9af480282aa31475f196737]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.751287 739 tablet_service.cc:1467] Processing CreateTablet for tablet 2484885df0da46f381d625a26e211c1e (DEFAULT_TABLE table=default.Table_2a410c48ef764ba2ac337b8471f11ba1 [id=c29c4105b83d4df9b617997c0c9c5b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.751600 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2484885df0da46f381d625a26e211c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.753358 1162 tablet_bootstrap.cc:492] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:57.755945 882 tablet_service.cc:1467] Processing CreateTablet for tablet 92aeca346a7746468dbbff053d933472 (DEFAULT_TABLE table=default.Table_db31d0a74f474c2c8d12130b143b1371 [id=60ff7f730d394990a0263e322d28d7ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.756186 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a059e2f0654a4aafe9e8082b8fde7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.756258 1162 tablet_bootstrap.cc:654] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.756848 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07a000ab73dd46ac8e134784a156040a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.757113 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83415f1c07ad495cabd287bdcd54dad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.757472 1165 ts_tablet_manager.cc:1918] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.757580 1165 log.cc:1198] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/07a63a0a8bb44fe9be0740fc777e610c
I20250115 14:51:57.757978 1165 ts_tablet_manager.cc:1939] T 07a63a0a8bb44fe9be0740fc777e610c P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.759047 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 07a63a0a8bb44fe9be0740fc777e610c (table default.Table_2a2f7c950304406484083d56f0b3177e [id=685971a2443b4b16a9af9f3a57d132c8]) successfully deleted
I20250115 14:51:57.759414 1165 tablet_replica.cc:331] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.759621 1165 raft_consensus.cc:2238] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.759832 1165 raft_consensus.cc:2267] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.760639 1165 ts_tablet_manager.cc:1905] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.762146 1164 tablet_bootstrap.cc:654] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.771415 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 24751e606ad94675b650d67a3738481e (table default.Table_49eb379e719f4677b13269d15ef7efad [id=af78a0b6494d4fd68224992dec097051]) successfully deleted
W20250115 14:51:57.771700 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4abde784f9c948cb96e6cc93c30817bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4abde784f9c948cb96e6cc93c30817bd already in progress: creating tablet
I20250115 14:51:57.772264 1165 ts_tablet_manager.cc:1918] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:57.772411 1165 log.cc:1198] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/2711a7d437f44b2fa11504d8e400e360
I20250115 14:51:57.772835 1165 ts_tablet_manager.cc:1939] T 2711a7d437f44b2fa11504d8e400e360 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.773794 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 2711a7d437f44b2fa11504d8e400e360 (table default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8 [id=a02ee91f154e41fd9ee54b476fd77331]) successfully deleted
I20250115 14:51:57.774196 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 90c1602aef514dd29d5d2cffb043dbb6
I20250115 14:51:57.775110 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c1602aef514dd29d5d2cffb043dbb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.775436 1162 tablet_bootstrap.cc:492] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:57.775553 1162 ts_tablet_manager.cc:1397] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.005s
I20250115 14:51:57.776099 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7628ff855c7f4396a10a9fb5e7a9acba" } new_table_name: "default.Table_e209cc3424eb4349b5387a32ac96d6bc" modify_external_catalogs: true
I20250115 14:51:57.776314 1162 raft_consensus.cc:357] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.776507 1162 raft_consensus.cc:383] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.776590 1162 raft_consensus.cc:738] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.777879 1628 raft_consensus.cc:491] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.777788 1162 consensus_queue.cc:260] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.778041 1628 raft_consensus.cc:513] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:57.778731 1628 leader_election.cc:290] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:57.780166 1162 ts_tablet_manager.cc:1428] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
W20250115 14:51:57.780241 710 leader_election.cc:343] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: bb792ca1b3f54def92f504ec8682d2c7
I20250115 14:51:57.780440 1162 tablet_bootstrap.cc:492] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:57.781670 984 consensus_peers.cc:487] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:57.781980 1162 tablet_bootstrap.cc:654] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.782881 1165 tablet_replica.cc:331] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:57.783071 1165 raft_consensus.cc:2238] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:57.783303 1165 pending_rounds.cc:62] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:57.783365 1165 pending_rounds.cc:69] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558331144609792 op_type: NO_OP noop_request { }
I20250115 14:51:57.783488 1165 raft_consensus.cc:2883] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:57.783581 1165 raft_consensus.cc:2267] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.784395 1165 ts_tablet_manager.cc:1905] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.785418 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d195be2d543f4a8693045a024912c43a
I20250115 14:51:57.786196 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7a794e73718b4f888c401b3038c977d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:57.786635 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet d195be2d543f4a8693045a024912c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:57.786759 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a794e73718b4f888c401b3038c977d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.791548 1165 ts_tablet_manager.cc:1918] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:57.791671 1165 log.cc:1198] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d86075ae5b894d5986ac4bd538f4d42a
I20250115 14:51:57.792047 1165 ts_tablet_manager.cc:1939] T d86075ae5b894d5986ac4bd538f4d42a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:57.793038 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d86075ae5b894d5986ac4bd538f4d42a (table default.Table_c5f3c652f95240ff9f080165af17f5e6 [id=54e9cab363084972abe02521b2cc19b8]) successfully deleted
I20250115 14:51:57.797127 888 tablet_service.cc:1467] Processing CreateTablet for tablet f06544890cbf460cbe218690e998eb1d (DEFAULT_TABLE table=default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.798913 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84844b5f50af408ab7dda73f7421cd9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.800506 1691 tablet.cc:1721] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:51:57.801870 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 4abde784f9c948cb96e6cc93c30817bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4abde784f9c948cb96e6cc93c30817bd already in progress: creating tablet
W20250115 14:51:57.806982 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 90c1602aef514dd29d5d2cffb043dbb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90c1602aef514dd29d5d2cffb043dbb6 already in progress: creating tablet
I20250115 14:51:57.807798 744 tablet_service.cc:1514] Processing DeleteTablet for tablet d195be2d543f4a8693045a024912c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:57.809523 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d195be2d543f4a8693045a024912c43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d195be2d543f4a8693045a024912c43a already in progress: creating tablet
I20250115 14:51:57.816359 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 07a000ab73dd46ac8e134784a156040a
I20250115 14:51:57.817070 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet caa1827366a34f039fd12deb855f07b6
I20250115 14:51:57.817250 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a000ab73dd46ac8e134784a156040a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.817831 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet caa1827366a34f039fd12deb855f07b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.817757 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6b336b36c2e2438fbf8698e281702756"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.818311 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b336b36c2e2438fbf8698e281702756 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:57.818388 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 07a000ab73dd46ac8e134784a156040a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a000ab73dd46ac8e134784a156040a already in progress: creating tablet
W20250115 14:51:57.818969 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet caa1827366a34f039fd12deb855f07b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caa1827366a34f039fd12deb855f07b6 already in progress: creating tablet
I20250115 14:51:57.819483 453 master_service.cc:946] ReplaceTablet: received request to replace tablet d71a1ecd900942e5ae788c9a28c70a7a from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:57.819730 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 0291acf45d75421b8b1c6311560e7f0c from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:57.820183 448 master_service.cc:946] ReplaceTablet: received request to replace tablet e198b4e1a65b4704bf2bb8f7d5987385 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:57.821321 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 9d8c53fdf16243f0a614e48ec4ac94f4 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:57.821532 756 tablet_service.cc:1514] Processing DeleteTablet for tablet bb792ca1b3f54def92f504ec8682d2c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:57.821699 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 104282a09c024bbaafeb5f28f91dadcb consensus state has no leader
I20250115 14:51:57.821935 1166 tablet_replica.cc:331] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.822132 1166 raft_consensus.cc:2238] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.822047 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bb95e5cd8bf543b79d33f685016648dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:57.822321 1166 raft_consensus.cc:2267] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:57.822513 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb95e5cd8bf543b79d33f685016648dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.823127 1166 ts_tablet_manager.cc:1905] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.823590 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f8c81f47cd6e4a308cd62124db38d42f
I20250115 14:51:57.824051 446 catalog_manager.cc:6596] ReplaceTablet: tablet f8c81f47cd6e4a308cd62124db38d42f deleted and replaced by tablet 580742eb5dac4dba9995809890e0e315
I20250115 14:51:57.824374 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c81f47cd6e4a308cd62124db38d42f with delete_type TABLET_DATA_DELETED (replaced by tablet 580742eb5dac4dba9995809890e0e315) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.824368 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6ccf0c2af49e4f2daf7ada5c0243d0ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.824829 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ccf0c2af49e4f2daf7ada5c0243d0ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:57.825234 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f8c81f47cd6e4a308cd62124db38d42f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8c81f47cd6e4a308cd62124db38d42f already in progress: creating tablet
W20250115 14:51:57.825922 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 consensus state has no leader
I20250115 14:51:57.826229 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_874111e557c1400bb8fb21c733becad1" } new_table_name: "default.Table_dbb5dd7ec474409b9743a74e69cc1b44" modify_external_catalogs: true
I20250115 14:51:57.826647 1162 tablet_bootstrap.cc:492] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:57.826821 1162 ts_tablet_manager.cc:1397] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.010s
I20250115 14:51:57.829913 1166 ts_tablet_manager.cc:1918] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.830109 1166 log.cc:1198] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/bb792ca1b3f54def92f504ec8682d2c7
I20250115 14:51:57.830682 1166 ts_tablet_manager.cc:1939] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.830104 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c1602aef514dd29d5d2cffb043dbb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.833027 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a000ab73dd46ac8e134784a156040a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.834050 744 tablet_service.cc:1514] Processing DeleteTablet for tablet fad6f9f7f853418e85353edfd38cc80e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.834239 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f312f1e3a33a404ca7284c713859f156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.835497 1164 tablet_bootstrap.cc:492] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.835636 1164 ts_tablet_manager.cc:1397] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.013s
I20250115 14:51:57.836061 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 31d95c9359e54f1a88a07d5c5cce7fa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.836426 1164 raft_consensus.cc:357] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.836613 1164 raft_consensus.cc:383] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.836697 1164 raft_consensus.cc:738] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.836886 1164 consensus_queue.cc:260] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.837481 1164 ts_tablet_manager.cc:1428] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:57.837754 1164 tablet_bootstrap.cc:492] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.839303 1164 tablet_bootstrap.cc:654] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:57.842664 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d195be2d543f4a8693045a024912c43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d195be2d543f4a8693045a024912c43a already in progress: creating tablet
I20250115 14:51:57.842892 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet bb792ca1b3f54def92f504ec8682d2c7 (table default.Table_76ab87d8a84c47008813112669e3800b [id=e1e34c5d139f4810a8f45a223e459e56]) successfully deleted
I20250115 14:51:57.843029 756 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c81f47cd6e4a308cd62124db38d42f with delete_type TABLET_DATA_DELETED (replaced by tablet 580742eb5dac4dba9995809890e0e315) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:57.843107 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 90c1602aef514dd29d5d2cffb043dbb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90c1602aef514dd29d5d2cffb043dbb6 already in progress: creating tablet
W20250115 14:51:57.843315 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 07a000ab73dd46ac8e134784a156040a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a000ab73dd46ac8e134784a156040a already in progress: creating tablet
I20250115 14:51:57.828156 1162 raft_consensus.cc:357] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.843725 1162 raft_consensus.cc:383] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.843825 1162 raft_consensus.cc:738] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.844707 1162 consensus_queue.cc:260] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.845057 1166 tablet_replica.cc:331] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
W20250115 14:51:57.845062 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f1ce59e05054a009cd36e12db35d449 has no consensus state
I20250115 14:51:57.845276 1162 ts_tablet_manager.cc:1428] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20250115 14:51:57.846555 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_69b643bd59a847959f3f29e23f007498"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:57.845324 1166 raft_consensus.cc:2238] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.847054 1166 raft_consensus.cc:2267] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:57.847106 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69b643bd59a847959f3f29e23f007498 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.847647 1162 tablet_bootstrap.cc:492] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:57.847869 1166 ts_tablet_manager.cc:1905] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.849063 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_eb6649c5d48f43e5959aa73d981a61cc" } new_table_name: "default.Table_498abe3411a94a3d8bbbf3c96571396c" modify_external_catalogs: true
I20250115 14:51:57.849350 1162 tablet_bootstrap.cc:654] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.848050 739 tablet_service.cc:1514] Processing DeleteTablet for tablet caa1827366a34f039fd12deb855f07b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.862841 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d71a1ecd900942e5ae788c9a28c70a7a
I20250115 14:51:57.863354 453 catalog_manager.cc:6596] ReplaceTablet: tablet d71a1ecd900942e5ae788c9a28c70a7a deleted and replaced by tablet 0651be54e0b244a4bace9f47efc9b58a
I20250115 14:51:57.865459 1164 tablet_bootstrap.cc:492] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.865790 1164 ts_tablet_manager.cc:1397] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.003s
I20250115 14:51:57.867043 1164 raft_consensus.cc:357] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.867359 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet d71a1ecd900942e5ae788c9a28c70a7a with delete_type TABLET_DATA_DELETED (replaced by tablet 0651be54e0b244a4bace9f47efc9b58a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:57.867383 1164 raft_consensus.cc:383] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.867502 1164 raft_consensus.cc:738] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.867888 1164 consensus_queue.cc:260] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.868860 1164 ts_tablet_manager.cc:1428] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:57.869065 1653 tablet.cc:1721] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:57.870270 1164 tablet_bootstrap.cc:492] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:51:57.871857 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5a059e2f0654a4aafe9e8082b8fde7a consensus state has no leader
W20250115 14:51:57.872045 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d71a1ecd900942e5ae788c9a28c70a7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d71a1ecd900942e5ae788c9a28c70a7a already in progress: creating tablet
I20250115 14:51:57.878993 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e198b4e1a65b4704bf2bb8f7d5987385
I20250115 14:51:57.879429 448 catalog_manager.cc:6596] ReplaceTablet: tablet e198b4e1a65b4704bf2bb8f7d5987385 deleted and replaced by tablet 65a6f83730f74d13823bd15d93538df7
I20250115 14:51:57.880187 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_875d7bd9588344008279ca89c1ad52be" } modify_external_catalogs: true
I20250115 14:51:57.880542 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3ffdd3389c224ce9a51ab9c816721b4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:57.881381 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_875d7bd9588344008279ca89c1ad52be" } modify_external_catalogs: true
I20250115 14:51:57.882002 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6279b757afda46ec908c5b3e5dd3a5c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:57.882683 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6279b757afda46ec908c5b3e5dd3a5c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:57.883363 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 282d988bc41443c381bb0ba69b618f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.883446 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet e198b4e1a65b4704bf2bb8f7d5987385 with delete_type TABLET_DATA_DELETED (replaced by tablet 65a6f83730f74d13823bd15d93538df7) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:57.883541 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ffdd3389c224ce9a51ab9c816721b4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:57.884194 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e198b4e1a65b4704bf2bb8f7d5987385 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e198b4e1a65b4704bf2bb8f7d5987385 already in progress: creating tablet
I20250115 14:51:57.884456 1164 tablet_bootstrap.cc:654] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:57.886359 710 leader_election.cc:343] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: bb792ca1b3f54def92f504ec8682d2c7
I20250115 14:51:57.886502 710 leader_election.cc:304] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:57.886720 710 raft_consensus.cc:2681] T bb792ca1b3f54def92f504ec8682d2c7 P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:51:57.887804 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fd72a0d586d4a2c817f6a39daa07063 with delete_type TABLET_DATA_DELETED (replaced by tablet d7309ecb11d844848638c042c9bbe5f1) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.890751 756 tablet_service.cc:1514] Processing DeleteTablet for tablet e198b4e1a65b4704bf2bb8f7d5987385 with delete_type TABLET_DATA_DELETED (replaced by tablet 65a6f83730f74d13823bd15d93538df7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.894222 756 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0c1c829aff438a94bc41609bdb7841 with delete_type TABLET_DATA_DELETED (replaced by tablet 23cba5cc51d7414a9352caedcc362c9c) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.897295 1166 ts_tablet_manager.cc:1918] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.899591 881 tablet_service.cc:1467] Processing CreateTablet for tablet 2484885df0da46f381d625a26e211c1e (DEFAULT_TABLE table=default.Table_2a410c48ef764ba2ac337b8471f11ba1 [id=c29c4105b83d4df9b617997c0c9c5b14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.900399 1166 log.cc:1198] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/fad6f9f7f853418e85353edfd38cc80e
I20250115 14:51:57.898347 1652 tablet.cc:1721] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:57.900885 1166 ts_tablet_manager.cc:1939] T fad6f9f7f853418e85353edfd38cc80e P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.902127 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet fad6f9f7f853418e85353edfd38cc80e (table default.Table_fb7d4709dd544378b8802e3294d6e441 [id=0d2a48bbde7045e69429c5464157ab08]) successfully deleted
I20250115 14:51:57.902485 1166 tablet_replica.cc:331] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.902709 1166 raft_consensus.cc:2238] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.902927 1166 raft_consensus.cc:2267] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.903448 909 raft_consensus.cc:1270] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:57.907063 1166 ts_tablet_manager.cc:1905] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.907541 1164 tablet_bootstrap.cc:492] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:57.907775 1164 ts_tablet_manager.cc:1397] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.003s
I20250115 14:51:57.903728 1168 raft_consensus.cc:357] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.909137 1164 raft_consensus.cc:357] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.909487 1164 raft_consensus.cc:383] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.909665 1164 raft_consensus.cc:738] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.909786 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 377d711f052545ddbe499afe7ba3de28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.910084 1164 consensus_queue.cc:260] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.911039 1164 ts_tablet_manager.cc:1428] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:57.911514 1164 tablet_bootstrap.cc:492] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:57.911583 883 tablet_service.cc:1514] Processing DeleteTablet for tablet d195be2d543f4a8693045a024912c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.913532 1540 consensus_queue.cc:1035] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:57.914268 1540 consensus_queue.cc:1035] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:57.914333 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 58cca6072e4e4f608c6a45be279304f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:57.914921 1164 tablet_bootstrap.cc:654] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:57.916649 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a000ab73dd46ac8e134784a156040a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:57.916877 847 consensus_peers.cc:487] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07a63a0a8bb44fe9be0740fc777e610c. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:57.919726 883 tablet_service.cc:1514] Processing DeleteTablet for tablet caa1827366a34f039fd12deb855f07b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.920002 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c1602aef514dd29d5d2cffb043dbb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.921638 883 tablet_service.cc:1514] Processing DeleteTablet for tablet d71a1ecd900942e5ae788c9a28c70a7a with delete_type TABLET_DATA_DELETED (replaced by tablet 0651be54e0b244a4bace9f47efc9b58a) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.921867 1628 raft_consensus.cc:491] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.922022 1628 raft_consensus.cc:513] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.922662 1628 leader_election.cc:290] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.923146 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f06544890cbf460cbe218690e998eb1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.927239 1168 raft_consensus.cc:383] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:57.927366 1168 raft_consensus.cc:738] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:57.927431 877 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c81f47cd6e4a308cd62124db38d42f with delete_type TABLET_DATA_DELETED (replaced by tablet 580742eb5dac4dba9995809890e0e315) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:57.929642 1166 ts_tablet_manager.cc:1918] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.929842 1166 log.cc:1198] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f312f1e3a33a404ca7284c713859f156
I20250115 14:51:57.930244 1166 ts_tablet_manager.cc:1939] T f312f1e3a33a404ca7284c713859f156 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.932055 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f312f1e3a33a404ca7284c713859f156 (table default.Table_2752e5f3a4ca4c41a614738b9354d962 [id=1c779c8a53f54707abe1d619b30a912e]) successfully deleted
I20250115 14:51:57.932636 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7309ecb11d844848638c042c9bbe5f1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:57.933125 1166 tablet_replica.cc:331] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.933387 1166 raft_consensus.cc:2238] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.933588 1166 raft_consensus.cc:2267] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.934413 909 raft_consensus.cc:1270] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:57.934456 1166 ts_tablet_manager.cc:1905] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.937338 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6942d102069d4c33b23fa2c0b29f94a6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:57.939477 848 consensus_peers.cc:487] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:57.939854 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 259ac764ca844df9b6c0ce1688a188da (table default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]) successfully deleted
W20250115 14:51:57.939924 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe696560ba6b4a76ad013d9a3486e398 consensus state has no leader
I20250115 14:51:57.940275 877 tablet_service.cc:1514] Processing DeleteTablet for tablet e198b4e1a65b4704bf2bb8f7d5987385 with delete_type TABLET_DATA_DELETED (replaced by tablet 65a6f83730f74d13823bd15d93538df7) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:57.940917 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f47a8acc3bec46b78725dcc3e675ea74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f47a8acc3bec46b78725dcc3e675ea74 already in progress: creating tablet
W20250115 14:51:57.941131 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 23cba5cc51d7414a9352caedcc362c9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23cba5cc51d7414a9352caedcc362c9c already in progress: creating tablet
I20250115 14:51:57.927213 875 tablet_service.cc:1467] Processing CreateTablet for tablet 95718d39a7824517a36c49c3b86fef30 (DEFAULT_TABLE table=default.Table_5d8548b30284461086fdcd351454ad57 [id=d114031263ea4c87a6c4d12c9d6a0d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:57.945590 1166 ts_tablet_manager.cc:1918] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.947127 1166 log.cc:1198] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/31d95c9359e54f1a88a07d5c5cce7fa1
I20250115 14:51:57.947573 1166 ts_tablet_manager.cc:1939] T 31d95c9359e54f1a88a07d5c5cce7fa1 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.948537 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 31d95c9359e54f1a88a07d5c5cce7fa1 (table default.Table_9984f5b933e4431ba6201377446f3026 [id=db3a3335dd864c7384b2a414c31a6d2b]) successfully deleted
W20250115 14:51:57.952755 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f8c81f47cd6e4a308cd62124db38d42f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8c81f47cd6e4a308cd62124db38d42f already in progress: creating tablet
W20250115 14:51:57.953379 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet caa1827366a34f039fd12deb855f07b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caa1827366a34f039fd12deb855f07b6 already in progress: creating tablet
I20250115 14:51:57.953712 1166 tablet_replica.cc:331] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.953929 1166 raft_consensus.cc:2238] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.954133 1166 raft_consensus.cc:2267] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.956815 1166 ts_tablet_manager.cc:1905] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.961557 1697 tablet.cc:1721] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250115 14:51:57.961941 848 consensus_peers.cc:487] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:51:57.962143 848 leader_election.cc:343] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:57.962266 848 leader_election.cc:304] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:57.965329 710 leader_election.cc:343] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 3fd72a0d586d4a2c817f6a39daa07063
I20250115 14:51:57.965960 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2a11b7b0dd6494daf7b017edd4317a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.966156 1663 raft_consensus.cc:2743] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:57.966224 1168 consensus_queue.cc:260] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.966765 1168 ts_tablet_manager.cc:1428] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.283s user 0.000s sys 0.002s
I20250115 14:51:57.968065 1628 raft_consensus.cc:491] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:57.968232 1628 raft_consensus.cc:513] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:57.968922 1628 leader_election.cc:290] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:57.969331 1166 ts_tablet_manager.cc:1918] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.969523 1166 log.cc:1198] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/3fd72a0d586d4a2c817f6a39daa07063
I20250115 14:51:57.969658 1168 tablet_bootstrap.cc:492] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:57.970007 1166 ts_tablet_manager.cc:1939] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:51:57.970669 710 leader_election.cc:343] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 58cca6072e4e4f608c6a45be279304f8
I20250115 14:51:57.970908 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 3fd72a0d586d4a2c817f6a39daa07063 (table default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]) successfully deleted
W20250115 14:51:57.975178 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e198b4e1a65b4704bf2bb8f7d5987385 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e198b4e1a65b4704bf2bb8f7d5987385 already in progress: creating tablet
I20250115 14:51:57.975559 1166 tablet_replica.cc:331] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.975801 1166 raft_consensus.cc:2238] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.975992 1166 raft_consensus.cc:2267] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.976635 1691 tablet.cc:1721] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:57.976751 1166 ts_tablet_manager.cc:1905] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:57.979106 1653 tablet.cc:1721] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:57.982483 1166 ts_tablet_manager.cc:1918] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:57.982933 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95718d39a7824517a36c49c3b86fef30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.983896 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2484885df0da46f381d625a26e211c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.990617 1166 log.cc:1198] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/fe0c1c829aff438a94bc41609bdb7841
I20250115 14:51:57.991021 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92aeca346a7746468dbbff053d933472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:57.991065 1166 ts_tablet_manager.cc:1939] T fe0c1c829aff438a94bc41609bdb7841 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:57.992007 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet fe0c1c829aff438a94bc41609bdb7841 (table default.Table_27cd4d793c8f4e77aa531c06dfa74a2d [id=a6994f260e1a4eada78d8c9e9fcf334e]) successfully deleted
I20250115 14:51:57.992377 1166 tablet_replica.cc:331] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:57.992875 1166 raft_consensus.cc:2238] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:57.993119 1166 raft_consensus.cc:2267] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:57.993865 1166 ts_tablet_manager.cc:1905] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.001024 739 tablet_service.cc:1514] Processing DeleteTablet for tablet d71a1ecd900942e5ae788c9a28c70a7a with delete_type TABLET_DATA_DELETED (replaced by tablet 0651be54e0b244a4bace9f47efc9b58a) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:58.001659 984 consensus_peers.cc:487] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:58.005484 447 catalog_manager.cc:5526] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:58.006656 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 6b60ff80679a470cb4bdced71cc14195 (table default.Table_c088519ba27c457983f341363f8a109e [id=201c214e2eab4873af2095e83dc88b23]). Expected version 2 got 0
I20250115 14:51:58.012066 1016 tablet_service.cc:1467] Processing CreateTablet for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 (DEFAULT_TABLE table=default.Table_3ef67fd49f0b4888a1f78b8e3b4558f0 [id=9e8ce51e784e4bdbb3ac78cba10bfedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.012562 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:58.001094 848 consensus_peers.cc:487] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:58.016589 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 97b599c6b8174959b8b66e90e9242057 (DEFAULT_TABLE table=default.Table_e3e5361d9cb144999d02ff4df3e1999b [id=5063a3c3ef3a4b1794a2c8532ab7adf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.017037 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b599c6b8174959b8b66e90e9242057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.020627 877 tablet_service.cc:1467] Processing CreateTablet for tablet 742792be08444792892bd9cd7336dda3 (DEFAULT_TABLE table=default.Table_97c066067f7a49cab2dd5bdade9f6de2 [id=8c1c9324a0cb4ce5b5f64af989e4bdf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.020601 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 742792be08444792892bd9cd7336dda3 (DEFAULT_TABLE table=default.Table_97c066067f7a49cab2dd5bdade9f6de2 [id=8c1c9324a0cb4ce5b5f64af989e4bdf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.020988 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742792be08444792892bd9cd7336dda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.021579 1653 tablet.cc:1721] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:51:58.020540 885 tablet_service.cc:1467] Processing CreateTablet for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 (DEFAULT_TABLE table=default.Table_3ef67fd49f0b4888a1f78b8e3b4558f0 [id=9e8ce51e784e4bdbb3ac78cba10bfedf]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:58.022029 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7233ebf3bae1462d8eeb3669b39944c0 consensus state has no leader
I20250115 14:51:58.024560 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1ce59e05054a009cd36e12db35d449 (DEFAULT_TABLE table=default.Table_e209cc3424eb4349b5387a32ac96d6bc [id=18a209f910d44ef081b5925cd6ebe028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.024900 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1ce59e05054a009cd36e12db35d449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.027336 1162 tablet_bootstrap.cc:492] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:58.027473 1162 ts_tablet_manager.cc:1397] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.180s user 0.003s sys 0.000s
I20250115 14:51:58.028285 1162 raft_consensus.cc:357] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.028489 1162 raft_consensus.cc:383] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.028590 1162 raft_consensus.cc:738] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.028851 1162 consensus_queue.cc:260] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.029687 1162 ts_tablet_manager.cc:1428] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:58.029961 1162 tablet_bootstrap.cc:492] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:58.030311 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.031030 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8e068b869e304185abaccf2d63685cd3" } modify_external_catalogs: true
I20250115 14:51:58.031222 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8e068b869e304185abaccf2d63685cd3" } modify_external_catalogs: true
I20250115 14:51:58.031941 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742792be08444792892bd9cd7336dda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.030098 874 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1ce59e05054a009cd36e12db35d449 (DEFAULT_TABLE table=default.Table_e209cc3424eb4349b5387a32ac96d6bc [id=18a209f910d44ef081b5925cd6ebe028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.033186 1168 tablet_bootstrap.cc:654] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.033314 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_44a23f5fdfb446a59f551a2a7a4ca2ce" } modify_external_catalogs: true
I20250115 14:51:58.034085 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_44a23f5fdfb446a59f551a2a7a4ca2ce" } modify_external_catalogs: true
I20250115 14:51:58.039372 739 tablet_service.cc:1467] Processing CreateTablet for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 (DEFAULT_TABLE table=default.Table_3ef67fd49f0b4888a1f78b8e3b4558f0 [id=9e8ce51e784e4bdbb3ac78cba10bfedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.039666 744 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1ce59e05054a009cd36e12db35d449 (DEFAULT_TABLE table=default.Table_e209cc3424eb4349b5387a32ac96d6bc [id=18a209f910d44ef081b5925cd6ebe028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.039819 756 tablet_service.cc:1467] Processing CreateTablet for tablet 97b599c6b8174959b8b66e90e9242057 (DEFAULT_TABLE table=default.Table_e3e5361d9cb144999d02ff4df3e1999b [id=5063a3c3ef3a4b1794a2c8532ab7adf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.040119 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.040273 748 tablet_service.cc:1467] Processing CreateTablet for tablet 742792be08444792892bd9cd7336dda3 (DEFAULT_TABLE table=default.Table_97c066067f7a49cab2dd5bdade9f6de2 [id=8c1c9324a0cb4ce5b5f64af989e4bdf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.040676 740 tablet_service.cc:1467] Processing CreateTablet for tablet ff9b57b8b9434f55ace5a4f47380df54 (DEFAULT_TABLE table=default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.042073 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a584af21c70545b8a2d3690c20422f1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.042623 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a584af21c70545b8a2d3690c20422f1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.043737 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4cfdadf8899b4ab9b5400953ec24e992" } modify_external_catalogs: true
I20250115 14:51:58.043885 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4cfdadf8899b4ab9b5400953ec24e992" } modify_external_catalogs: true
I20250115 14:51:58.045147 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_56a365aae805419683d1f4dfd97951cc" } modify_external_catalogs: true
I20250115 14:51:58.045315 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_56a365aae805419683d1f4dfd97951cc" } modify_external_catalogs: true
I20250115 14:51:58.046931 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b599c6b8174959b8b66e90e9242057. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.052589 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742792be08444792892bd9cd7336dda3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.054972 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1ce59e05054a009cd36e12db35d449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.056886 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff9b57b8b9434f55ace5a4f47380df54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.058228 1231 raft_consensus.cc:491] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.058382 1231 raft_consensus.cc:513] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.059051 1231 leader_election.cc:290] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.059636 1697 tablet.cc:1721] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:58.059788 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1ce59e05054a009cd36e12db35d449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.059806 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83097c5bfe364418a213236435edc40c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.059935 887 tablet_service.cc:1467] Processing CreateTablet for tablet 97b599c6b8174959b8b66e90e9242057 (DEFAULT_TABLE table=default.Table_e3e5361d9cb144999d02ff4df3e1999b [id=5063a3c3ef3a4b1794a2c8532ab7adf6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:58.060387 984 leader_election.cc:343] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:58.065099 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 151967us
I20250115 14:51:58.066838 1540 consensus_queue.cc:1035] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:51:58.066934 710 leader_election.cc:343] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 3fd72a0d586d4a2c817f6a39daa07063
I20250115 14:51:58.067093 710 leader_election.cc:304] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:58.067283 710 raft_consensus.cc:2681] T 3fd72a0d586d4a2c817f6a39daa07063 P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250115 14:51:58.067973 710 leader_election.cc:343] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 58cca6072e4e4f608c6a45be279304f8
I20250115 14:51:58.068099 710 leader_election.cc:304] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:58.068356 710 raft_consensus.cc:2681] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:51:58.072647 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_a8abdc319d2c413ba113c127fbfd5971 [id=33e3df28f2d445c5ba4e8b0ce4458c1a] alter complete (version 1)
I20250115 14:51:58.073201 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9f7b9d2bd72741fbb3132278c8a77d1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.073836 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f7b9d2bd72741fbb3132278c8a77d1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.074615 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9d8c53fdf16243f0a614e48ec4ac94f4
I20250115 14:51:58.075111 449 catalog_manager.cc:6596] ReplaceTablet: tablet 9d8c53fdf16243f0a614e48ec4ac94f4 deleted and replaced by tablet 3b82924e735846b2914520684d1b8fc7
I20250115 14:51:58.076272 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b82924e735846b2914520684d1b8fc7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.076587 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c0d7ba16034c431d8c39b3ca49d63d88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.077243 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d8c53fdf16243f0a614e48ec4ac94f4 already in progress: creating tablet
I20250115 14:51:58.077309 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 75788f8eb36b446e927197f624fbfdf5
I20250115 14:51:58.078891 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 75788f8eb36b446e927197f624fbfdf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.080080 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d810e40dae3d42e38f38ebf32e62305e" } new_table_name: "default.Table_8e6652dab3bc46b7b4f1b66415398217" modify_external_catalogs: true
W20250115 14:51:58.081269 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 75788f8eb36b446e927197f624fbfdf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75788f8eb36b446e927197f624fbfdf5 already in progress: creating tablet
I20250115 14:51:58.079488 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83097c5bfe364418a213236435edc40c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:51:58.081969 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0d7ba16034c431d8c39b3ca49d63d88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.083686 910 raft_consensus.cc:2463] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:58.087919 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4b90205ef15f470a99c013e7f2cf99c6 (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]). Expected version 3 got 0
W20250115 14:51:58.088249 984 consensus_peers.cc:487] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:58.089185 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5bb88f06405a4a91b7deda4c03d9ba5f" } new_table_name: "default.Table_98cb517939064897baf89d74e6e0355e" modify_external_catalogs: true
I20250115 14:51:58.090731 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b82924e735846b2914520684d1b8fc7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:58.094389 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 75788f8eb36b446e927197f624fbfdf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:58.095388 983 leader_election.cc:304] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.095714 1540 raft_consensus.cc:2798] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:58.095842 1540 raft_consensus.cc:491] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:58.095937 1540 raft_consensus.cc:3054] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:58.069554 1162 tablet_bootstrap.cc:654] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.097815 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 32eee3d3dfec464baf10c238babb9eb1 (DEFAULT_TABLE table=default.Table_c5c33df1942c4b90a8069108477e91f5 [id=31fde8bd2aa3452fb638c948eb5d8a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.097931 1012 tablet_service.cc:1467] Processing CreateTablet for tablet b5177265ba974ad6a381adb1e8c49589 (DEFAULT_TABLE table=default.Table_b6a520bf69264c44b1da30c0ffb41cc0 [id=f0b642abaec54506b98190fd1ac9ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.098230 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32eee3d3dfec464baf10c238babb9eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.098470 1017 tablet_service.cc:1467] Processing CreateTablet for tablet ec8c34919648447fac14208ede0ddc01 (DEFAULT_TABLE table=default.Table_fe77f8263ada4a8390e949b24c6ae559 [id=9f0b8730115147268be0c1fc0a4b7a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.098803 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8c34919648447fac14208ede0ddc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.100378 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b5937d6ad0494734a402fdc594eb9415 (DEFAULT_TABLE table=default.Table_c6bb6ec57a9240c48a9f5607865e62c2 [id=62ff1172217947669bf02f96134877c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.102808 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2d04209fd34918b83978325a5e1df8 (DEFAULT_TABLE table=default.Table_53f051fa3f6b4051882de44c9baff2aa [id=1907295f0b5c49b9b99a188d4dc9f7f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.102846 746 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2d04209fd34918b83978325a5e1df8 (DEFAULT_TABLE table=default.Table_53f051fa3f6b4051882de44c9baff2aa [id=1907295f0b5c49b9b99a188d4dc9f7f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.103264 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2d04209fd34918b83978325a5e1df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.103389 749 tablet_service.cc:1467] Processing CreateTablet for tablet ec8c34919648447fac14208ede0ddc01 (DEFAULT_TABLE table=default.Table_fe77f8263ada4a8390e949b24c6ae559 [id=9f0b8730115147268be0c1fc0a4b7a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.103745 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8c34919648447fac14208ede0ddc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.108394 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5177265ba974ad6a381adb1e8c49589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.108543 750 tablet_service.cc:1467] Processing CreateTablet for tablet 32eee3d3dfec464baf10c238babb9eb1 (DEFAULT_TABLE table=default.Table_c5c33df1942c4b90a8069108477e91f5 [id=31fde8bd2aa3452fb638c948eb5d8a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.108942 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32eee3d3dfec464baf10c238babb9eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.112578 741 tablet_service.cc:1467] Processing CreateTablet for tablet b5937d6ad0494734a402fdc594eb9415 (DEFAULT_TABLE table=default.Table_c6bb6ec57a9240c48a9f5607865e62c2 [id=62ff1172217947669bf02f96134877c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.113543 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5937d6ad0494734a402fdc594eb9415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.098805 1540 raft_consensus.cc:513] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.117550 1231 raft_consensus.cc:491] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.118054 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2d04209fd34918b83978325a5e1df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.118031 1231 raft_consensus.cc:513] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.118752 1027 tablet_service.cc:1467] Processing CreateTablet for tablet ff9b57b8b9434f55ace5a4f47380df54 (DEFAULT_TABLE table=default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.118830 1231 leader_election.cc:290] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.119921 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5937d6ad0494734a402fdc594eb9415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.120711 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff9b57b8b9434f55ace5a4f47380df54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.122351 1540 leader_election.cc:290] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.124140 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 75788f8eb36b446e927197f624fbfdf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.132501 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f960c88df14242c5ada1ecf9d8e68196
I20250115 14:51:58.133353 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_82a2ce9b462c48b28d9fa64f0f426ec8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.133941 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82a2ce9b462c48b28d9fa64f0f426ec8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.135596 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 8e6cc08065494d9e85eeff5044b44ba6
I20250115 14:51:58.136281 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2898e18fe4e64a1a92ed3658b91dc502" } new_table_name: "default.Table_ebc003502c644be98ae8fec069e7bb39" modify_external_catalogs: true
I20250115 14:51:58.138033 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_15773744107143d6b1e5215c832f0a3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.138516 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15773744107143d6b1e5215c832f0a3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.139967 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b2a11b7b0dd6494daf7b017edd4317a5
I20250115 14:51:58.140655 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_040aa3c43ac34524823fe14b90fe4b9c" } new_table_name: "default.Table_28a52958055546da964198b7c17f8b8c" modify_external_catalogs: true
I20250115 14:51:58.143208 739 tablet_service.cc:1514] Processing DeleteTablet for tablet f960c88df14242c5ada1ecf9d8e68196 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.143553 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.144539 739 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a11b7b0dd6494daf7b017edd4317a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.144667 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a73bd0141ccc4ec4b052b18b9fb62bb7
I20250115 14:51:58.145340 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a602da76783c41a89e586a035929a1ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.145859 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a602da76783c41a89e586a035929a1ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.148489 739 tablet_service.cc:1514] Processing DeleteTablet for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.151087 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet f960c88df14242c5ada1ecf9d8e68196 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.151365 1165 tablet_replica.cc:331] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:58.151577 1165 raft_consensus.cc:2238] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:58.151986 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a11b7b0dd6494daf7b017edd4317a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.116197 752 tablet_service.cc:1467] Processing CreateTablet for tablet b5177265ba974ad6a381adb1e8c49589 (DEFAULT_TABLE table=default.Table_b6a520bf69264c44b1da30c0ffb41cc0 [id=f0b642abaec54506b98190fd1ac9ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.153098 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5177265ba974ad6a381adb1e8c49589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.152995 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83097c5bfe364418a213236435edc40c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:58.153792 984 leader_election.cc:343] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.153932 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.154072 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.155109 1165 raft_consensus.cc:2267] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.155652 1165 ts_tablet_manager.cc:1905] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.161700 1165 ts_tablet_manager.cc:1918] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:58.161834 1165 log.cc:1198] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f960c88df14242c5ada1ecf9d8e68196
I20250115 14:51:58.162218 1165 ts_tablet_manager.cc:1939] T f960c88df14242c5ada1ecf9d8e68196 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:58.163945 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f960c88df14242c5ada1ecf9d8e68196 (table default.Table_56a365aae805419683d1f4dfd97951cc [id=24d4e203bce64fcdbd5b6fa30a99e6b2]) successfully deleted
I20250115 14:51:58.128118 1166 ts_tablet_manager.cc:1918] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:58.164662 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b2a11b7b0dd6494daf7b017edd4317a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a11b7b0dd6494daf7b017edd4317a5 already in progress: creating tablet
W20250115 14:51:58.165513 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 8e6cc08065494d9e85eeff5044b44ba6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e6cc08065494d9e85eeff5044b44ba6 already in progress: creating tablet
W20250115 14:51:58.166287 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a73bd0141ccc4ec4b052b18b9fb62bb7 already in progress: creating tablet
I20250115 14:51:58.168742 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ce445d334842c8a6bacad9f693feda" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:58.172190 984 leader_election.cc:343] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.174464 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task__Z8ae/build/asan/bin/kudu pid 494
I20250115 14:51:58.176306 556 generic_service.cc:196] Checking for leaks (request via RPC)
I20250115 14:51:58.180050 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b599c6b8174959b8b66e90e9242057. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:58.182670 848 consensus_peers.cc:487] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:58.187191 910 raft_consensus.cc:1270] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:51:58.188817 1166 log.cc:1198] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/58cca6072e4e4f608c6a45be279304f8
I20250115 14:51:58.189321 1166 ts_tablet_manager.cc:1939] T 58cca6072e4e4f608c6a45be279304f8 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:58.190647 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b82924e735846b2914520684d1b8fc7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.194485 891 tablet_service.cc:1467] Processing CreateTablet for tablet ff9b57b8b9434f55ace5a4f47380df54 (DEFAULT_TABLE table=default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.195658 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 58cca6072e4e4f608c6a45be279304f8 (table default.Table_ca27ab47d077491489383629eaaf1437 [id=5ec280e869eb4ba6bec54fbc22af1409]) successfully deleted
I20250115 14:51:58.195996 1167 ts_tablet_manager.cc:1918] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:58.197475 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ce445d334842c8a6bacad9f693feda" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:58.199129 1698 consensus_queue.cc:1035] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:51:58.198684 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d71a1ecd900942e5ae788c9a28c70a7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d71a1ecd900942e5ae788c9a28c70a7a already in progress: creating tablet
I20250115 14:51:58.199688 1166 tablet_replica.cc:331] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
W20250115 14:51:58.200332 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 75788f8eb36b446e927197f624fbfdf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75788f8eb36b446e927197f624fbfdf5 already in progress: creating tablet
I20250115 14:51:58.200644 1166 raft_consensus.cc:2238] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.201359 1166 raft_consensus.cc:2267] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.203167 1166 ts_tablet_manager.cc:1905] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.205687 1698 raft_consensus.cc:491] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.205858 1698 raft_consensus.cc:513] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.206597 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff9b57b8b9434f55ace5a4f47380df54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.206589 1698 leader_election.cc:290] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.207479 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83097c5bfe364418a213236435edc40c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:58.207808 1164 tablet_bootstrap.cc:492] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.207861 884 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2d04209fd34918b83978325a5e1df8 (DEFAULT_TABLE table=default.Table_53f051fa3f6b4051882de44c9baff2aa [id=1907295f0b5c49b9b99a188d4dc9f7f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.207955 1164 ts_tablet_manager.cc:1397] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.297s user 0.045s sys 0.102s
I20250115 14:51:58.208474 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e990a2920b542f69a8bdd022db6e7e2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:58.209249 984 leader_election.cc:343] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.209618 912 raft_consensus.cc:2463] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:58.209935 1164 raft_consensus.cc:357] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.210155 1164 raft_consensus.cc:383] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.210230 1164 raft_consensus.cc:738] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.210497 1164 consensus_queue.cc:260] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.210995 1164 ts_tablet_manager.cc:1428] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:58.211649 1164 tablet_bootstrap.cc:492] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.212381 1166 ts_tablet_manager.cc:1918] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:58.212729 878 tablet_service.cc:1467] Processing CreateTablet for tablet b5177265ba974ad6a381adb1e8c49589 (DEFAULT_TABLE table=default.Table_b6a520bf69264c44b1da30c0ffb41cc0 [id=f0b642abaec54506b98190fd1ac9ce67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.214411 1164 tablet_bootstrap.cc:654] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.215376 1162 tablet_bootstrap.cc:492] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:58.215559 1162 ts_tablet_manager.cc:1397] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.186s user 0.004s sys 0.001s
I20250115 14:51:58.216349 1166 log.cc:1198] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f960c88df14242c5ada1ecf9d8e68196
I20250115 14:51:58.216390 1162 raft_consensus.cc:357] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.216609 1162 raft_consensus.cc:383] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.216686 1162 raft_consensus.cc:738] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.216939 1162 consensus_queue.cc:260] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.217463 1166 ts_tablet_manager.cc:1939] T f960c88df14242c5ada1ecf9d8e68196 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:58.219552 1162 ts_tablet_manager.cc:1428] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250115 14:51:58.219846 1162 tablet_bootstrap.cc:492] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:58.219942 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_52bf98f0d5b44013ba19e15f0fc24cce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.220559 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52bf98f0d5b44013ba19e15f0fc24cce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.221449 1162 tablet_bootstrap.cc:654] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.221575 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_721928c380b840ad84f9f5030b52632f [id=c5f831177088497dbf1fee130e231e37] alter complete (version 1)
I20250115 14:51:58.222827 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_db31d0a74f474c2c8d12130b143b1371" } new_table_name: "default.Table_df2dd54e8d8749aa942872c4b20ef257" modify_external_catalogs: true
I20250115 14:51:58.224483 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_06fdf72400894754b2da4a05ae517fac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.224973 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06fdf72400894754b2da4a05ae517fac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.225586 910 raft_consensus.cc:3054] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:58.226598 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59d8fd4b26294e8ca6745b95f2e03231 consensus state has no leader
I20250115 14:51:58.227365 876 tablet_service.cc:1467] Processing CreateTablet for tablet 32eee3d3dfec464baf10c238babb9eb1 (DEFAULT_TABLE table=default.Table_c5c33df1942c4b90a8069108477e91f5 [id=31fde8bd2aa3452fb638c948eb5d8a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.228235 1698 raft_consensus.cc:491] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.228407 1698 raft_consensus.cc:513] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.229120 1698 leader_election.cc:290] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.228279 983 leader_election.cc:304] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.230474 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9e2c053d624b6ba71b031a591475a4" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.231457 1698 raft_consensus.cc:2798] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:58.232158 1698 raft_consensus.cc:491] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:58.232272 1698 raft_consensus.cc:3054] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:58.232609 984 leader_election.cc:343] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.234370 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f960c88df14242c5ada1ecf9d8e68196 (table default.Table_56a365aae805419683d1f4dfd97951cc [id=24d4e203bce64fcdbd5b6fa30a99e6b2]) successfully deleted
W20250115 14:51:58.234620 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 8e6cc08065494d9e85eeff5044b44ba6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e6cc08065494d9e85eeff5044b44ba6 already in progress: creating tablet
W20250115 14:51:58.234843 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b2a11b7b0dd6494daf7b017edd4317a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a11b7b0dd6494daf7b017edd4317a5 already in progress: creating tablet
I20250115 14:51:58.234848 1698 raft_consensus.cc:513] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:58.235037 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a73bd0141ccc4ec4b052b18b9fb62bb7 already in progress: creating tablet
W20250115 14:51:58.235209 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d8c53fdf16243f0a614e48ec4ac94f4 already in progress: creating tablet
I20250115 14:51:58.235461 1698 leader_election.cc:290] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.237001 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f104c1a71404591072e7a36b276b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b3f07ba1d929468eb8eb1f949e9931b9) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.237304 1166 tablet_replica.cc:331] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:58.237565 1166 raft_consensus.cc:2238] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.237867 1166 raft_consensus.cc:2267] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.238667 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ce445d334842c8a6bacad9f693feda" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:58.239180 984 leader_election.cc:343] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.243562 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ca710cea154be5b0f57ace4af4168b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.243844 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ce445d334842c8a6bacad9f693feda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.244035 1165 tablet_replica.cc:331] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:58.244307 1165 raft_consensus.cc:2238] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.244232 1652 tablet.cc:1721] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:58.244488 1165 raft_consensus.cc:2267] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.245100 1165 ts_tablet_manager.cc:1905] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.227284 1167 log.cc:1198] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/86f0144904ae4008945ccf325039dac1
I20250115 14:51:58.247309 1166 ts_tablet_manager.cc:1905] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.255545 893 tablet_service.cc:1467] Processing CreateTablet for tablet b5937d6ad0494734a402fdc594eb9415 (DEFAULT_TABLE table=default.Table_c6bb6ec57a9240c48a9f5607865e62c2 [id=62ff1172217947669bf02f96134877c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.257184 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5177265ba974ad6a381adb1e8c49589. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.257805 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_880a5165055d48ffba4f0508b2aebfdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.258423 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_880a5165055d48ffba4f0508b2aebfdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.260462 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d6182b9f2b0e4105887762b69f004053"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.260969 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6182b9f2b0e4105887762b69f004053 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.261191 1164 tablet_bootstrap.cc:492] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.261782 1164 ts_tablet_manager.cc:1397] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.006s
I20250115 14:51:58.262248 1162 tablet_bootstrap.cc:492] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:58.262388 1162 ts_tablet_manager.cc:1397] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.003s
I20250115 14:51:58.262616 1164 raft_consensus.cc:357] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.262830 1164 raft_consensus.cc:383] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.262916 1164 raft_consensus.cc:738] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.263168 1164 consensus_queue.cc:260] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.263686 1164 ts_tablet_manager.cc:1428] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:58.264870 879 tablet_service.cc:1514] Processing DeleteTablet for tablet f960c88df14242c5ada1ecf9d8e68196 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:58.266027 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5937d6ad0494734a402fdc594eb9415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.266350 875 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a11b7b0dd6494daf7b017edd4317a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:58.267988 1164 tablet_bootstrap.cc:492] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.268448 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e990a2920b542f69a8bdd022db6e7e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4ee8b63cc21145d8993b058d8c6d23c7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.268886 1167 ts_tablet_manager.cc:1939] T 86f0144904ae4008945ccf325039dac1 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.269757 1164 tablet_bootstrap.cc:654] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.271910 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:58.272827 1162 raft_consensus.cc:357] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.273016 1162 raft_consensus.cc:383] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.273087 1162 raft_consensus.cc:738] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.273310 1162 consensus_queue.cc:260] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.273895 1162 ts_tablet_manager.cc:1428] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20250115 14:51:58.274132 883 tablet_service.cc:1514] Processing DeleteTablet for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:58.274343 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 314) took 1283 ms. Trace:
I20250115 14:51:58.274426 1167 rpcz_store.cc:276] 0115 14:51:56.985288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:56.985342 (+ 54us) service_pool.cc:225] Handling call
0115 14:51:57.585059 (+599717us) ts_tablet_manager.cc:1145] Deleting tablet 86f0144904ae4008945ccf325039dac1
0115 14:51:57.585065 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:57.966755 (+381690us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.190075 (+223320us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.269201 (+ 79126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":599495}
I20250115 14:51:58.274760 1167 tablet_replica.cc:331] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.274937 1167 raft_consensus.cc:2238] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.275111 1167 raft_consensus.cc:2267] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.275610 1167 ts_tablet_manager.cc:1905] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.276250 1166 ts_tablet_manager.cc:1918] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.276378 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 86f0144904ae4008945ccf325039dac1 (table default.Table_39eb08eb474049f2a2fbe741ef730d68 [id=ab6f49066e884652b77a0b9a791575c7]) successfully deleted
I20250115 14:51:58.276563 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet d86075ae5b894d5986ac4bd538f4d42a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.276782 1162 tablet_bootstrap.cc:492] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:58.277750 1166 log.cc:1198] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ea0f104c1a71404591072e7a36b276b0
I20250115 14:51:58.278174 1166 ts_tablet_manager.cc:1939] T ea0f104c1a71404591072e7a36b276b0 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:58.279029 890 tablet_service.cc:1467] Processing CreateTablet for tablet ec8c34919648447fac14208ede0ddc01 (DEFAULT_TABLE table=default.Table_fe77f8263ada4a8390e949b24c6ae559 [id=9f0b8730115147268be0c1fc0a4b7a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.280144 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ea0f104c1a71404591072e7a36b276b0 (table default.Table_30d9bde332c24d2a94267ffd2605a7cf [id=717e74f1625e4a5a92c254f204df1629]) successfully deleted
I20250115 14:51:58.280706 1164 tablet_bootstrap.cc:492] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.278954 912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9e2c053d624b6ba71b031a591475a4" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:58.281320 1164 ts_tablet_manager.cc:1397] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20250115 14:51:58.282130 1164 raft_consensus.cc:357] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.282351 1164 raft_consensus.cc:383] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.282428 1164 raft_consensus.cc:738] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.282961 1165 ts_tablet_manager.cc:1918] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.282655 1164 consensus_queue.cc:260] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.283411 1164 ts_tablet_manager.cc:1428] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:51:58.283712 1164 tablet_bootstrap.cc:492] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.284960 1165 log.cc:1198] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/50ca710cea154be5b0f57ace4af4168b
I20250115 14:51:58.285349 1164 tablet_bootstrap.cc:654] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.284763 1162 tablet_bootstrap.cc:654] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:58.286629 847 consensus_peers.cc:487] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2711a7d437f44b2fa11504d8e400e360. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:58.286890 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b2d04209fd34918b83978325a5e1df8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.287528 1165 ts_tablet_manager.cc:1939] T 50ca710cea154be5b0f57ace4af4168b P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:58.288499 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec8c34919648447fac14208ede0ddc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.290843 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32eee3d3dfec464baf10c238babb9eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.296068 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 50ca710cea154be5b0f57ace4af4168b (table default.Table_d809e208790b4c1a911708b393be6a3f [id=d45a9a753a874c41a842f25450ab10d2]) successfully deleted
I20250115 14:51:58.296492 1165 tablet_replica.cc:331] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:58.296730 1165 raft_consensus.cc:2238] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.296958 1165 raft_consensus.cc:2267] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.297508 1165 ts_tablet_manager.cc:1905] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.303562 1165 ts_tablet_manager.cc:1918] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.303678 1165 log.cc:1198] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b9ce445d334842c8a6bacad9f693feda
I20250115 14:51:58.304016 1165 ts_tablet_manager.cc:1939] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:58.304764 1165 tablet_replica.cc:331] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:58.304739 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b9ce445d334842c8a6bacad9f693feda (table default.Table_ff3a0751d563485db14fc5a44116f2ac [id=4334f62c5f0a43119d21690c19315404]) successfully deleted
I20250115 14:51:58.304967 1165 raft_consensus.cc:2238] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.305137 1165 raft_consensus.cc:2267] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.305634 1165 ts_tablet_manager.cc:1905] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.306272 910 raft_consensus.cc:2463] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:58.308445 1162 tablet_bootstrap.cc:492] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:58.308588 1162 ts_tablet_manager.cc:1397] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20250115 14:51:58.309120 1164 tablet_bootstrap.cc:492] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.309265 1164 ts_tablet_manager.cc:1397] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250115 14:51:58.309342 1162 raft_consensus.cc:357] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.309552 1162 raft_consensus.cc:383] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.309615 1162 raft_consensus.cc:738] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.309844 1162 consensus_queue.cc:260] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.310307 1162 ts_tablet_manager.cc:1428] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:58.310817 1162 tablet_bootstrap.cc:492] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:58.310005 1164 raft_consensus.cc:357] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.311443 1164 raft_consensus.cc:383] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.311545 1164 raft_consensus.cc:738] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.311877 1164 consensus_queue.cc:260] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.282749 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9ce445d334842c8a6bacad9f693feda" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:58.312865 1164 ts_tablet_manager.cc:1428] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:51:58.313155 1164 tablet_bootstrap.cc:492] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.315760 906 raft_consensus.cc:3054] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:58.319259 1164 tablet_bootstrap.cc:654] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.319890 1165 ts_tablet_manager.cc:1918] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.320016 1165 log.cc:1198] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3e990a2920b542f69a8bdd022db6e7e2
I20250115 14:51:58.320365 1165 ts_tablet_manager.cc:1939] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:58.321560 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3e990a2920b542f69a8bdd022db6e7e2 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]) successfully deleted
I20250115 14:51:58.322202 1162 tablet_bootstrap.cc:654] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:58.332013 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d86075ae5b894d5986ac4bd538f4d42a because the tablet was not found. No further retry: Not found: Tablet not found: d86075ae5b894d5986ac4bd538f4d42a
I20250115 14:51:58.340018 1164 tablet_bootstrap.cc:492] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.340196 1164 ts_tablet_manager.cc:1397] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.000s
I20250115 14:51:58.340972 1164 raft_consensus.cc:357] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.341202 1164 raft_consensus.cc:383] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.341279 1164 raft_consensus.cc:738] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.341544 1164 consensus_queue.cc:260] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.342067 1164 ts_tablet_manager.cc:1428] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:58.346215 1164 tablet_bootstrap.cc:492] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.349660 1164 tablet_bootstrap.cc:654] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.364372 1164 tablet_bootstrap.cc:492] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.364527 1164 ts_tablet_manager.cc:1397] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
W20250115 14:51:58.365063 847 consensus_peers.cc:487] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e50462626f914fb1b0f440c48f30e72d. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:58.365267 1164 raft_consensus.cc:357] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.365494 1164 raft_consensus.cc:383] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.366083 1164 raft_consensus.cc:738] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.366344 1164 consensus_queue.cc:260] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.366869 1164 ts_tablet_manager.cc:1428] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:58.367440 1164 tablet_bootstrap.cc:492] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.367691 906 raft_consensus.cc:2463] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:58.371780 1164 tablet_bootstrap.cc:654] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.384270 1164 tablet_bootstrap.cc:492] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.384472 1164 ts_tablet_manager.cc:1397] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.005s
I20250115 14:51:58.385314 1164 raft_consensus.cc:357] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.385540 1164 raft_consensus.cc:383] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.385607 1164 raft_consensus.cc:738] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.385843 1164 consensus_queue.cc:260] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.386312 1164 ts_tablet_manager.cc:1428] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250115 14:51:58.386691 1164 tablet_bootstrap.cc:492] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.392810 1164 tablet_bootstrap.cc:654] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:58.402346 983 leader_election.cc:343] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 3e990a2920b542f69a8bdd022db6e7e2
I20250115 14:51:58.402556 983 leader_election.cc:304] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:58.402880 983 raft_consensus.cc:2681] T 3e990a2920b542f69a8bdd022db6e7e2 P c238c037a1c74725a001e2b4a1ff4304: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250115 14:51:58.403923 983 leader_election.cc:343] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.404155 983 leader_election.cc:304] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.404558 1698 raft_consensus.cc:2743] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.404963 983 leader_election.cc:304] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.405295 1673 raft_consensus.cc:2798] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:58.405424 1673 raft_consensus.cc:695] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:58.405526 983 leader_election.cc:304] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.405606 1164 tablet_bootstrap.cc:492] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.405742 1164 ts_tablet_manager.cc:1397] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
I20250115 14:51:58.405741 1673 consensus_queue.cc:237] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.406530 1164 raft_consensus.cc:357] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.406734 1164 raft_consensus.cc:383] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.406828 1164 raft_consensus.cc:738] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.407073 1164 consensus_queue.cc:260] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.407740 1164 ts_tablet_manager.cc:1428] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:58.408069 1164 tablet_bootstrap.cc:492] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.409198 1167 ts_tablet_manager.cc:1918] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.409463 1167 log.cc:1198] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/1e7580b6245b41628874d9dcbc3366e2
I20250115 14:51:58.410143 1167 ts_tablet_manager.cc:1939] T 1e7580b6245b41628874d9dcbc3366e2 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.410759 1164 tablet_bootstrap.cc:654] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.410696 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 315) took 1425 ms. Trace:
I20250115 14:51:58.411021 1167 rpcz_store.cc:276] 0115 14:51:56.985378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:56.985424 (+ 46us) service_pool.cc:225] Handling call
0115 14:51:58.274680 (+1289256us) ts_tablet_manager.cc:1145] Deleting tablet 1e7580b6245b41628874d9dcbc3366e2
0115 14:51:58.274690 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.360349 (+ 85659us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.408939 (+ 48590us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.410639 (+ 1700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1288714}
I20250115 14:51:58.411767 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 1e7580b6245b41628874d9dcbc3366e2 (table default.Table_843fb099548148f29e7b622a4ad34b61 [id=085d5f79fc6c4bdb87752ecc55d89445]) successfully deleted
I20250115 14:51:58.412540 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 313) took 1443 ms. Trace:
I20250115 14:51:58.426508 1700 tablet.cc:1721] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:58.435436 1164 tablet_bootstrap.cc:492] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.435621 1164 ts_tablet_manager.cc:1397] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250115 14:51:58.437791 1164 raft_consensus.cc:357] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.438031 1164 raft_consensus.cc:383] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.438104 1164 raft_consensus.cc:738] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.438329 1164 consensus_queue.cc:260] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.438828 1164 ts_tablet_manager.cc:1428] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:58.439143 1164 tablet_bootstrap.cc:492] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.440752 1164 tablet_bootstrap.cc:654] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.455727 1698 raft_consensus.cc:491] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.455994 1698 raft_consensus.cc:513] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.457129 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50edca843e54f4b8e803d4d180b736d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:58.457789 984 leader_election.cc:343] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.458263 1698 leader_election.cc:290] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:58.459656 983 raft_consensus.cc:2681] T b9ce445d334842c8a6bacad9f693feda P c238c037a1c74725a001e2b4a1ff4304: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:51:58.473069 775 raft_consensus.cc:3054] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:58.479445 1167 rpcz_store.cc:276] 0115 14:51:56.968881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:56.971905 (+ 3024us) service_pool.cc:225] Handling call
0115 14:51:58.412244 (+1440339us) ts_tablet_manager.cc:1145] Deleting tablet d97c2f443ab040f9bc1924f45379165e
0115 14:51:58.412255 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.412484 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406124}
W20250115 14:51:58.484915 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 79403.2us
W20250115 14:51:58.486095 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d97c2f443ab040f9bc1924f45379165e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d97c2f443ab040f9bc1924f45379165e already in progress: creating tablet
I20250115 14:51:58.461038 1168 tablet_bootstrap.cc:492] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.486689 1168 ts_tablet_manager.cc:1397] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.520s user 0.014s sys 0.010s
I20250115 14:51:58.487659 1168 raft_consensus.cc:357] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.487993 1168 raft_consensus.cc:383] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.488126 1168 raft_consensus.cc:738] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.488516 1168 consensus_queue.cc:260] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.489308 1168 ts_tablet_manager.cc:1428] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250115 14:51:58.489655 1624 raft_consensus.cc:491] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.489784 1624 raft_consensus.cc:513] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.490485 1624 leader_election.cc:290] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:58.491362 1168 tablet_bootstrap.cc:492] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:58.491442 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ca710cea154be5b0f57ace4af4168b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.491886 1167 tablet_replica.cc:331] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
W20250115 14:51:58.492151 847 leader_election.cc:343] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 50ca710cea154be5b0f57ace4af4168b
W20250115 14:51:58.500999 848 leader_election.cc:343] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.501430 848 leader_election.cc:304] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.501873 1663 raft_consensus.cc:2743] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.503302 1167 raft_consensus.cc:2238] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:58.504570 1700 tablet.cc:1721] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:58.505316 1653 tablet.cc:1721] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:58.505944 1167 pending_rounds.cc:62] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:58.510581 1652 tablet.cc:1721] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:58.507911 1167 pending_rounds.cc:69] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558327802445824 op_type: NO_OP noop_request { }
I20250115 14:51:58.515803 1167 raft_consensus.cc:2883] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:58.516943 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50edca843e54f4b8e803d4d180b736d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:58.517385 909 raft_consensus.cc:2463] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:58.518492 1167 raft_consensus.cc:2267] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.519917 1652 tablet.cc:1721] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:58.521446 1168 tablet_bootstrap.cc:654] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.530977 1162 tablet_bootstrap.cc:492] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:58.531941 1652 tablet.cc:1721] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:58.531137 1162 ts_tablet_manager.cc:1397] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.220s user 0.076s sys 0.063s
I20250115 14:51:58.533663 1162 raft_consensus.cc:357] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.533891 1162 raft_consensus.cc:383] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.533989 1162 raft_consensus.cc:738] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.534210 1162 consensus_queue.cc:260] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.534744 1162 ts_tablet_manager.cc:1428] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:58.534727 1167 ts_tablet_manager.cc:1905] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.534997 1231 raft_consensus.cc:491] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.535082 1162 tablet_bootstrap.cc:492] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:58.535180 1231 raft_consensus.cc:513] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.535857 1231 leader_election.cc:290] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.536720 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b5bc2c71cf41809d7057acd47e27be" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:58.537364 984 leader_election.cc:343] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.537583 1731 tablet.cc:1721] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:51:58.538549 983 leader_election.cc:304] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.540035 1168 tablet_bootstrap.cc:492] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.542501 1164 tablet_bootstrap.cc:492] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.542661 1164 ts_tablet_manager.cc:1397] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.104s user 0.022s sys 0.063s
I20250115 14:51:58.542661 1698 raft_consensus.cc:2798] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:58.542917 1168 ts_tablet_manager.cc:1397] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20250115 14:51:58.543550 1164 raft_consensus.cc:357] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.543802 1164 raft_consensus.cc:383] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.543905 1164 raft_consensus.cc:738] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.544145 1164 consensus_queue.cc:260] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.544692 1164 ts_tablet_manager.cc:1428] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:58.544986 1164 tablet_bootstrap.cc:492] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.543673 1168 raft_consensus.cc:357] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.545533 1628 raft_consensus.cc:491] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.545677 1628 raft_consensus.cc:513] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.546336 1628 leader_election.cc:290] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.546679 1164 tablet_bootstrap.cc:654] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.546885 1168 raft_consensus.cc:383] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:51:58.547930 710 leader_election.cc:343] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 745153c106844076bb020f9ed026c97a
I20250115 14:51:58.551419 1698 raft_consensus.cc:491] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:58.551910 1698 raft_consensus.cc:3054] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:58.554574 1164 tablet_bootstrap.cc:492] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.554754 1164 ts_tablet_manager.cc:1397] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250115 14:51:58.554821 1168 raft_consensus.cc:738] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.556015 1164 raft_consensus.cc:357] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.556232 1164 raft_consensus.cc:383] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.556356 1164 raft_consensus.cc:738] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.556525 1168 consensus_queue.cc:260] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.556582 1164 consensus_queue.cc:260] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.557092 1164 ts_tablet_manager.cc:1428] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:58.557323 1168 ts_tablet_manager.cc:1428] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250115 14:51:58.557402 1164 tablet_bootstrap.cc:492] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.557754 1168 tablet_bootstrap.cc:492] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:58.559002 1164 tablet_bootstrap.cc:654] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.559461 1168 tablet_bootstrap.cc:654] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.563294 1162 tablet_bootstrap.cc:654] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.563235 1698 raft_consensus.cc:513] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.564141 1653 tablet.cc:1716] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: Skipping requested alter to schema version 2, tablet already version 2
W20250115 14:51:58.564421 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 68734us
I20250115 14:51:58.567332 1698 leader_election.cc:290] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.567708 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50edca843e54f4b8e803d4d180b736d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:58.568405 984 leader_election.cc:343] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.572906 1168 tablet_bootstrap.cc:492] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.573123 1168 ts_tablet_manager.cc:1397] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20250115 14:51:58.573239 1167 ts_tablet_manager.cc:1918] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:58.573369 1167 log.cc:1198] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/07a63a0a8bb44fe9be0740fc777e610c
I20250115 14:51:58.573762 1167 ts_tablet_manager.cc:1939] T 07a63a0a8bb44fe9be0740fc777e610c P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.574298 1168 raft_consensus.cc:357] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.574767 1168 raft_consensus.cc:383] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.574911 1168 raft_consensus.cc:738] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.574525 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 318) took 1572 ms. Trace:
I20250115 14:51:58.575193 1167 rpcz_store.cc:276] 0115 14:51:57.001678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.001832 (+ 154us) service_pool.cc:225] Handling call
0115 14:51:58.491814 (+1489982us) ts_tablet_manager.cc:1145] Deleting tablet 07a63a0a8bb44fe9be0740fc777e610c
0115 14:51:58.491827 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.570616 (+ 78789us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.573107 (+ 2491us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.574491 (+ 1384us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1484893}
I20250115 14:51:58.575510 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b5bc2c71cf41809d7057acd47e27be" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:58.575757 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 07a63a0a8bb44fe9be0740fc777e610c (table default.Table_2a2f7c950304406484083d56f0b3177e [id=685971a2443b4b16a9af9f3a57d132c8]) successfully deleted
I20250115 14:51:58.576146 1167 tablet_replica.cc:331] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.576448 1167 raft_consensus.cc:2238] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:58.576757 1167 pending_rounds.cc:62] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:58.576849 1167 pending_rounds.cc:69] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558329388949504 op_type: NO_OP noop_request { }
I20250115 14:51:58.576992 1167 raft_consensus.cc:2883] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:58.577095 1167 raft_consensus.cc:2267] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.577702 1167 ts_tablet_manager.cc:1905] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.578876 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50edca843e54f4b8e803d4d180b736d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:58.579064 909 raft_consensus.cc:3054] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:58.580461 1168 consensus_queue.cc:260] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.581601 1168 ts_tablet_manager.cc:1428] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:51:58.581915 1168 tablet_bootstrap.cc:492] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:58.582001 909 raft_consensus.cc:2463] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:58.584033 1652 tablet.cc:1716] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: Skipping requested alter to schema version 2, tablet already version 2
W20250115 14:51:58.585155 710 leader_election.cc:343] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 745153c106844076bb020f9ed026c97a
I20250115 14:51:58.585295 710 leader_election.cc:304] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:58.585706 1628 raft_consensus.cc:2743] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:58.586133 983 leader_election.cc:343] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.586779 983 leader_election.cc:304] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.588045 1231 raft_consensus.cc:2743] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.589792 1164 tablet_bootstrap.cc:492] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.589946 1164 ts_tablet_manager.cc:1397] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.010s
I20250115 14:51:58.590615 1162 tablet_bootstrap.cc:492] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:58.590828 1162 ts_tablet_manager.cc:1397] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.002s
I20250115 14:51:58.590729 1164 raft_consensus.cc:357] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.590934 1164 raft_consensus.cc:383] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.591022 1164 raft_consensus.cc:738] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.591241 1164 consensus_queue.cc:260] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.591786 1164 ts_tablet_manager.cc:1428] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:58.592103 1164 tablet_bootstrap.cc:492] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.592443 1162 raft_consensus.cc:357] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.592715 1162 raft_consensus.cc:383] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.592820 1162 raft_consensus.cc:738] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.593114 1162 consensus_queue.cc:260] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.593645 1162 ts_tablet_manager.cc:1428] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:58.593729 1164 tablet_bootstrap.cc:654] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.594005 1162 tablet_bootstrap.cc:492] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:58.596475 983 leader_election.cc:304] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.596838 1653 tablet.cc:1721] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:58.598765 1624 raft_consensus.cc:491] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.598939 1624 raft_consensus.cc:513] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.599617 1624 leader_election.cc:290] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:58.600374 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73bd0141ccc4ec4b052b18b9fb62bb7" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.601357 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a73bd0141ccc4ec4b052b18b9fb62bb7" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:58.602080 848 leader_election.cc:343] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:58.602514 847 leader_election.cc:343] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.602640 847 leader_election.cc:304] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.603094 1624 raft_consensus.cc:2743] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.603606 1168 tablet_bootstrap.cc:654] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.603849 1698 raft_consensus.cc:2798] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:58.604007 1698 raft_consensus.cc:695] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:58.604257 1698 consensus_queue.cc:237] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.605731 1162 tablet_bootstrap.cc:654] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.609303 1652 tablet.cc:1721] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:58.612741 1167 ts_tablet_manager.cc:1918] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:58.612870 1167 log.cc:1198] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e50462626f914fb1b0f440c48f30e72d
I20250115 14:51:58.613318 1167 ts_tablet_manager.cc:1939] T e50462626f914fb1b0f440c48f30e72d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.614204 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 320) took 1609 ms. Trace:
I20250115 14:51:58.614333 1167 rpcz_store.cc:276] 0115 14:51:57.004665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.005407 (+ 742us) service_pool.cc:225] Handling call
0115 14:51:58.576075 (+1570668us) ts_tablet_manager.cc:1145] Deleting tablet e50462626f914fb1b0f440c48f30e72d
0115 14:51:58.576085 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.592190 (+ 16105us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.612586 (+ 20396us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.614148 (+ 1562us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1568944}
I20250115 14:51:58.614873 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet e50462626f914fb1b0f440c48f30e72d (table default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]) successfully deleted
I20250115 14:51:58.616077 1168 tablet_bootstrap.cc:492] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.616240 1168 ts_tablet_manager.cc:1397] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250115 14:51:58.616990 1168 raft_consensus.cc:357] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.617257 1168 raft_consensus.cc:383] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.617322 1168 raft_consensus.cc:738] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.617538 1168 consensus_queue.cc:260] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.618144 1168 ts_tablet_manager.cc:1428] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:58.618438 1168 tablet_bootstrap.cc:492] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:58.620923 1168 tablet_bootstrap.cc:654] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:58.626430 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4294c49da1e044be993100b3cf5e2cea has no consensus state
I20250115 14:51:58.629726 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7a794e73718b4f888c401b3038c977d9" } modify_external_catalogs: true
I20250115 14:51:58.629932 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7a794e73718b4f888c401b3038c977d9" } modify_external_catalogs: true
I20250115 14:51:58.632711 1162 tablet_bootstrap.cc:492] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:58.632879 1162 ts_tablet_manager.cc:1397] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.003s
I20250115 14:51:58.633489 1027 tablet_service.cc:1467] Processing CreateTablet for tablet ca128bf3d6fc430298da932a78f54184 (DEFAULT_TABLE table=default.Table_3ffdd3389c224ce9a51ab9c816721b4b [id=6cdcb9c1b34642d8a792807746ac26f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.633638 1162 raft_consensus.cc:357] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.633834 1162 raft_consensus.cc:383] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.633916 1162 raft_consensus.cc:738] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.633946 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca128bf3d6fc430298da932a78f54184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.634184 876 tablet_service.cc:1467] Processing CreateTablet for tablet ca128bf3d6fc430298da932a78f54184 (DEFAULT_TABLE table=default.Table_3ffdd3389c224ce9a51ab9c816721b4b [id=6cdcb9c1b34642d8a792807746ac26f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.634155 1162 consensus_queue.cc:260] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.634568 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca128bf3d6fc430298da932a78f54184. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.634840 1162 ts_tablet_manager.cc:1428] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:58.635118 1162 tablet_bootstrap.cc:492] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:58.636548 890 tablet_service.cc:1467] Processing CreateTablet for tablet dc363f70f3f44d309f69fac9356c6257 (DEFAULT_TABLE table=default.Table_6279b757afda46ec908c5b3e5dd3a5c2 [id=1752ebbf59404ebebaba188a4e01f461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.636819 1628 raft_consensus.cc:491] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.636879 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc363f70f3f44d309f69fac9356c6257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.636967 1628 raft_consensus.cc:513] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.637604 1628 leader_election.cc:290] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.638252 1017 tablet_service.cc:1467] Processing CreateTablet for tablet ba596cfb3ce64718a42a9083975edc8d (DEFAULT_TABLE table=default.Table_6ccf0c2af49e4f2daf7ada5c0243d0ca [id=4d98519fb7194978a2a1533d7e42f2f1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:58.638592 710 leader_election.cc:343] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: f55c75e5febb407398b0bc06a1db9b1a
I20250115 14:51:58.638701 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba596cfb3ce64718a42a9083975edc8d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:58.639329 710 leader_election.cc:343] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: f55c75e5febb407398b0bc06a1db9b1a
I20250115 14:51:58.639457 710 leader_election.cc:304] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:58.639816 1628 raft_consensus.cc:2743] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.639917 1024 tablet_service.cc:1467] Processing CreateTablet for tablet dc363f70f3f44d309f69fac9356c6257 (DEFAULT_TABLE table=default.Table_6279b757afda46ec908c5b3e5dd3a5c2 [id=1752ebbf59404ebebaba188a4e01f461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.640568 893 tablet_service.cc:1467] Processing CreateTablet for tablet ba596cfb3ce64718a42a9083975edc8d (DEFAULT_TABLE table=default.Table_6ccf0c2af49e4f2daf7ada5c0243d0ca [id=4d98519fb7194978a2a1533d7e42f2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.640916 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba596cfb3ce64718a42a9083975edc8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.641081 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc363f70f3f44d309f69fac9356c6257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.644423 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c088519ba27c457983f341363f8a109e" } modify_external_catalogs: true
I20250115 14:51:58.644609 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c088519ba27c457983f341363f8a109e" } modify_external_catalogs: true
I20250115 14:51:58.648893 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f104c1a71404591072e7a36b276b0 with delete_type TABLET_DATA_DELETED (replaced by tablet b3f07ba1d929468eb8eb1f949e9931b9) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.649349 756 tablet_service.cc:1467] Processing CreateTablet for tablet ca128bf3d6fc430298da932a78f54184 (DEFAULT_TABLE table=default.Table_3ffdd3389c224ce9a51ab9c816721b4b [id=6cdcb9c1b34642d8a792807746ac26f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.649767 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca128bf3d6fc430298da932a78f54184. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:58.650701 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ea0f104c1a71404591072e7a36b276b0 because the tablet was not found. No further retry: Not found: Tablet not found: ea0f104c1a71404591072e7a36b276b0
I20250115 14:51:58.652431 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 51b5bc2c71cf41809d7057acd47e27be (table default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]). Expected version 1 got 0
I20250115 14:51:58.652580 750 tablet_service.cc:1467] Processing CreateTablet for tablet dc363f70f3f44d309f69fac9356c6257 (DEFAULT_TABLE table=default.Table_6279b757afda46ec908c5b3e5dd3a5c2 [id=1752ebbf59404ebebaba188a4e01f461]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.652827 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]). Expected version 1 got 0
I20250115 14:51:58.652937 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc363f70f3f44d309f69fac9356c6257. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.654914 1162 tablet_bootstrap.cc:654] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.656601 740 tablet_service.cc:1467] Processing CreateTablet for tablet ba596cfb3ce64718a42a9083975edc8d (DEFAULT_TABLE table=default.Table_6ccf0c2af49e4f2daf7ada5c0243d0ca [id=4d98519fb7194978a2a1533d7e42f2f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.656977 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba596cfb3ce64718a42a9083975edc8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.664887 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]). Expected version 2 got 0
I20250115 14:51:58.666534 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 1839edd0d49b467d8bb31b3001ff89eb (table default.Table_253ac9c181914d0f891baf1f8722dbd0 [id=a724f6e4ad16485b8ee2445bb072be3e]). Expected version 1 got 0
I20250115 14:51:58.668238 752 tablet_service.cc:1467] Processing CreateTablet for tablet 076278b53996407c8a5c73a389459eaa (DEFAULT_TABLE table=default.Table_7a794e73718b4f888c401b3038c977d9 [id=2b96b49ab0d64a7db19e0fb7985b7bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.668699 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 076278b53996407c8a5c73a389459eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.674181 754 tablet_service.cc:1467] Processing CreateTablet for tablet 580742eb5dac4dba9995809890e0e315 (DEFAULT_TABLE table=default.Table_8a7055866139444681c82d83d9af5978 [id=b6de0dbbced44c83af5f98a9d1020cfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.674247 741 tablet_service.cc:1467] Processing CreateTablet for tablet b803d8e103a5423f85c75c9006809b17 (DEFAULT_TABLE table=default.Table_4dff2e869f8141e09dd757d6e47118f4 [id=43852683a5a64e1b85403395fa3d210c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.674654 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580742eb5dac4dba9995809890e0e315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.674801 746 tablet_service.cc:1467] Processing CreateTablet for tablet 4294c49da1e044be993100b3cf5e2cea (DEFAULT_TABLE table=default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.674844 887 tablet_service.cc:1467] Processing CreateTablet for tablet 076278b53996407c8a5c73a389459eaa (DEFAULT_TABLE table=default.Table_7a794e73718b4f888c401b3038c977d9 [id=2b96b49ab0d64a7db19e0fb7985b7bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.675216 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 076278b53996407c8a5c73a389459eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.675163 744 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce498ffd3f04c5d8eb642739a838903 (DEFAULT_TABLE table=default.Table_6b336b36c2e2438fbf8698e281702756 [id=c91058c4abd64f2fa0dadce63f6ce0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.675519 739 tablet_service.cc:1467] Processing CreateTablet for tablet 3b82924e735846b2914520684d1b8fc7 (DEFAULT_TABLE table=default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.675976 749 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7e70288eb248f1a71646c66e376a63 (DEFAULT_TABLE table=default.Table_bb95e5cd8bf543b79d33f685016648dd [id=13d5cd4ccf2245319f40495141007b58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.676352 743 tablet_service.cc:1467] Processing CreateTablet for tablet 65a6f83730f74d13823bd15d93538df7 (DEFAULT_TABLE table=default.Table_7488025a06ba42b5814d9788890cf2a0 [id=a3bf28673dac4ddab579709d6fd8885d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.676790 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0651be54e0b244a4bace9f47efc9b58a (DEFAULT_TABLE table=default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.677055 748 tablet_service.cc:1467] Processing CreateTablet for tablet a6f3f8cae6244968927b864e7ce97565 (DEFAULT_TABLE table=default.Table_66c26e1f10144558b65efccffc30c61f [id=02f9f84c80334c36801ef8e760120ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.677409 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4294c49da1e044be993100b3cf5e2cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.679440 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce498ffd3f04c5d8eb642739a838903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.680585 874 tablet_service.cc:1467] Processing CreateTablet for tablet b803d8e103a5423f85c75c9006809b17 (DEFAULT_TABLE table=default.Table_4dff2e869f8141e09dd757d6e47118f4 [id=43852683a5a64e1b85403395fa3d210c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.680943 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b803d8e103a5423f85c75c9006809b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.684559 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b82924e735846b2914520684d1b8fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.684576 885 tablet_service.cc:1467] Processing CreateTablet for tablet a6f3f8cae6244968927b864e7ce97565 (DEFAULT_TABLE table=default.Table_66c26e1f10144558b65efccffc30c61f [id=02f9f84c80334c36801ef8e760120ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.684921 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f3f8cae6244968927b864e7ce97565. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.688514 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7e70288eb248f1a71646c66e376a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.688658 884 tablet_service.cc:1467] Processing CreateTablet for tablet 4294c49da1e044be993100b3cf5e2cea (DEFAULT_TABLE table=default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.689074 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4294c49da1e044be993100b3cf5e2cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.689074 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a6f83730f74d13823bd15d93538df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.689888 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0651be54e0b244a4bace9f47efc9b58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.690330 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f3f8cae6244968927b864e7ce97565. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.690847 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b803d8e103a5423f85c75c9006809b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.692618 878 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce498ffd3f04c5d8eb642739a838903 (DEFAULT_TABLE table=default.Table_6b336b36c2e2438fbf8698e281702756 [id=c91058c4abd64f2fa0dadce63f6ce0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.692986 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce498ffd3f04c5d8eb642739a838903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.696579 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3b82924e735846b2914520684d1b8fc7 (DEFAULT_TABLE table=default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.696933 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b82924e735846b2914520684d1b8fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.701807 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 076278b53996407c8a5c73a389459eaa (DEFAULT_TABLE table=default.Table_7a794e73718b4f888c401b3038c977d9 [id=2b96b49ab0d64a7db19e0fb7985b7bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.701968 1020 tablet_service.cc:1467] Processing CreateTablet for tablet a6f3f8cae6244968927b864e7ce97565 (DEFAULT_TABLE table=default.Table_66c26e1f10144558b65efccffc30c61f [id=02f9f84c80334c36801ef8e760120ab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.702255 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 076278b53996407c8a5c73a389459eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.702561 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce498ffd3f04c5d8eb642739a838903 (DEFAULT_TABLE table=default.Table_6b336b36c2e2438fbf8698e281702756 [id=c91058c4abd64f2fa0dadce63f6ce0f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.702673 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 65a6f83730f74d13823bd15d93538df7 (DEFAULT_TABLE table=default.Table_7488025a06ba42b5814d9788890cf2a0 [id=a3bf28673dac4ddab579709d6fd8885d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.702759 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 4294c49da1e044be993100b3cf5e2cea (DEFAULT_TABLE table=default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.702916 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ce498ffd3f04c5d8eb642739a838903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.703359 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a6f83730f74d13823bd15d93538df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.703846 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 3b82924e735846b2914520684d1b8fc7 (DEFAULT_TABLE table=default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.704476 886 tablet_service.cc:1467] Processing CreateTablet for tablet 0651be54e0b244a4bace9f47efc9b58a (DEFAULT_TABLE table=default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.704520 1628 raft_consensus.cc:491] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.704679 1628 raft_consensus.cc:513] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.705335 1628 leader_election.cc:290] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.705798 1027 tablet_service.cc:1467] Processing CreateTablet for tablet b803d8e103a5423f85c75c9006809b17 (DEFAULT_TABLE table=default.Table_4dff2e869f8141e09dd757d6e47118f4 [id=43852683a5a64e1b85403395fa3d210c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.706061 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7e70288eb248f1a71646c66e376a63 (DEFAULT_TABLE table=default.Table_bb95e5cd8bf543b79d33f685016648dd [id=13d5cd4ccf2245319f40495141007b58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.706521 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0651be54e0b244a4bace9f47efc9b58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.706547 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 0651be54e0b244a4bace9f47efc9b58a (DEFAULT_TABLE table=default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.707284 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 580742eb5dac4dba9995809890e0e315 (DEFAULT_TABLE table=default.Table_8a7055866139444681c82d83d9af5978 [id=b6de0dbbced44c83af5f98a9d1020cfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.707419 892 tablet_service.cc:1467] Processing CreateTablet for tablet 580742eb5dac4dba9995809890e0e315 (DEFAULT_TABLE table=default.Table_8a7055866139444681c82d83d9af5978 [id=b6de0dbbced44c83af5f98a9d1020cfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.707767 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580742eb5dac4dba9995809890e0e315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.708117 1698 raft_consensus.cc:491] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.708251 1698 raft_consensus.cc:513] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.709029 1698 leader_election.cc:290] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.709226 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3ec000089a47a2930dec382192cd6f" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:58.709255 882 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7e70288eb248f1a71646c66e376a63 (DEFAULT_TABLE table=default.Table_bb95e5cd8bf543b79d33f685016648dd [id=13d5cd4ccf2245319f40495141007b58]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.709352 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aad920dd9744d9b87a8234c92232d8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.709538 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7e70288eb248f1a71646c66e376a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.710251 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4294c49da1e044be993100b3cf5e2cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.709918 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3ec000089a47a2930dec382192cd6f" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:58.710546 877 tablet_service.cc:1467] Processing CreateTablet for tablet 65a6f83730f74d13823bd15d93538df7 (DEFAULT_TABLE table=default.Table_7488025a06ba42b5814d9788890cf2a0 [id=a3bf28673dac4ddab579709d6fd8885d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.711143 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6f3f8cae6244968927b864e7ce97565. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.711210 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65a6f83730f74d13823bd15d93538df7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:58.711163 984 leader_election.cc:343] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.711627 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c7e70288eb248f1a71646c66e376a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.711798 710 leader_election.cc:304] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:58.712244 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b803d8e103a5423f85c75c9006809b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.712390 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aad920dd9744d9b87a8234c92232d8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:51:58.712965 983 leader_election.cc:343] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.713092 983 leader_election.cc:304] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.713296 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0651be54e0b244a4bace9f47efc9b58a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.713946 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 580742eb5dac4dba9995809890e0e315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.714183 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b82924e735846b2914520684d1b8fc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.714684 1628 raft_consensus.cc:2743] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.727259 1628 raft_consensus.cc:491] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.727429 1628 raft_consensus.cc:513] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.728124 1628 leader_election.cc:290] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:58.729511 710 leader_election.cc:343] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: a6b76cb343fb4456ac07fd3c6e9888de
I20250115 14:51:58.758554 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task__Z8ae/build/asan/bin/kudu with pid 494
I20250115 14:51:58.761324 749 tablet_service.cc:1467] Processing CreateTablet for tablet a48227ec5aa846c3ac63bd29ed38be25 (DEFAULT_TABLE table=default.Table_69b643bd59a847959f3f29e23f007498 [id=924019d3b4e842c2b3f173b307885151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.761775 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48227ec5aa846c3ac63bd29ed38be25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.761967 1698 raft_consensus.cc:491] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.762676 882 tablet_service.cc:1467] Processing CreateTablet for tablet a48227ec5aa846c3ac63bd29ed38be25 (DEFAULT_TABLE table=default.Table_69b643bd59a847959f3f29e23f007498 [id=924019d3b4e842c2b3f173b307885151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.763034 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48227ec5aa846c3ac63bd29ed38be25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.774223 1628 raft_consensus.cc:491] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.774405 1628 raft_consensus.cc:513] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.774994 1628 leader_election.cc:290] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:58.776675 710 leader_election.cc:343] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: b027233cfd314d0ebda472b4588d3f00
I20250115 14:51:58.780488 1167 tablet_replica.cc:331] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.780758 1167 raft_consensus.cc:2238] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.780933 1167 raft_consensus.cc:2267] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.781519 1167 ts_tablet_manager.cc:1905] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.785382 1164 tablet_bootstrap.cc:492] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:58.785557 1164 ts_tablet_manager.cc:1397] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.194s user 0.018s sys 0.014s
I20250115 14:51:58.786440 1164 raft_consensus.cc:357] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.786669 1164 raft_consensus.cc:383] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.786756 1164 raft_consensus.cc:738] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.787020 1164 consensus_queue.cc:260] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.787528 1164 ts_tablet_manager.cc:1428] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:58.787868 1164 tablet_bootstrap.cc:492] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:58.792312 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c156b9383c644939a6ada0bb7d7cf235" } modify_external_catalogs: true
I20250115 14:51:58.792544 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c156b9383c644939a6ada0bb7d7cf235" } modify_external_catalogs: true
W20250115 14:51:58.793927 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92aeca346a7746468dbbff053d933472 consensus state has no leader
I20250115 14:51:58.794736 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_21fa1a9206d64c8bb19a405f37518a65" } modify_external_catalogs: true
I20250115 14:51:58.794920 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_21fa1a9206d64c8bb19a405f37518a65" } modify_external_catalogs: true
I20250115 14:51:58.795615 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_17b896568dc34b1aaf2b3e197d03fdc4" } modify_external_catalogs: true
I20250115 14:51:58.795763 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_17b896568dc34b1aaf2b3e197d03fdc4" } modify_external_catalogs: true
I20250115 14:51:58.797152 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 50273a0019bb4cd8977f7ef014372d85 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:58.800577 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_538a75b4e6e54b619c8a011996f4f971"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.801100 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_538a75b4e6e54b619c8a011996f4f971 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.802820 1698 raft_consensus.cc:513] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:58.803220 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 49294.4us
I20250115 14:51:58.803604 1698 leader_election.cc:290] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.804366 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f847f92cd122423aa93f1afa701b0bbb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:58.805759 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f847f92cd122423aa93f1afa701b0bbb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.806694 1698 raft_consensus.cc:491] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.806843 1698 raft_consensus.cc:513] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.807382 1164 tablet_bootstrap.cc:654] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.807698 1167 ts_tablet_manager.cc:1918] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.807824 1167 log.cc:1198] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9aeca9975f1945ad956f47db1ea16a73
I20250115 14:51:58.808653 1167 ts_tablet_manager.cc:1939] T 9aeca9975f1945ad956f47db1ea16a73 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.813587 1168 tablet_bootstrap.cc:492] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.813747 1168 ts_tablet_manager.cc:1397] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.195s user 0.005s sys 0.000s
I20250115 14:51:58.813887 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 076278b53996407c8a5c73a389459eaa
I20250115 14:51:58.814590 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_872859e534b64345b7989072c4d38b8f" } new_table_name: "default.Table_f129f5835a304494a797ad1eb37cf8fe" modify_external_catalogs: true
I20250115 14:51:58.814500 1168 raft_consensus.cc:357] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.814862 1168 raft_consensus.cc:383] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.814954 1168 raft_consensus.cc:738] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.815114 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 076278b53996407c8a5c73a389459eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:58.815200 1168 consensus_queue.cc:260] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.815896 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6b60ff80679a470cb4bdced71cc14195
I20250115 14:51:58.816582 1168 ts_tablet_manager.cc:1428] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:58.816602 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2061d08104dd455788c1d0e8c792abc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:58.817144 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2061d08104dd455788c1d0e8c792abc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.817193 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b60ff80679a470cb4bdced71cc14195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:58.817345 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 319) took 1812 ms. Trace:
I20250115 14:51:58.817410 1168 tablet_bootstrap.cc:492] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:58.817440 1167 rpcz_store.cc:276] 0115 14:51:57.004529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.005377 (+ 848us) service_pool.cc:225] Handling call
0115 14:51:58.780406 (+1775029us) ts_tablet_manager.cc:1145] Deleting tablet 9aeca9975f1945ad956f47db1ea16a73
0115 14:51:58.780418 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.796241 (+ 15823us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.807573 (+ 11332us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.817281 (+ 9708us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773148}
I20250115 14:51:58.817862 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 317) took 1832 ms. Trace:
I20250115 14:51:58.817943 1167 rpcz_store.cc:276] 0115 14:51:56.985850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:56.985893 (+ 43us) service_pool.cc:225] Handling call
0115 14:51:58.817694 (+1831801us) ts_tablet_manager.cc:1145] Deleting tablet b2c385edf4414157845b971aaf366831
0115 14:51:58.817703 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.817807 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794129}
I20250115 14:51:58.818022 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9aeca9975f1945ad956f47db1ea16a73 (table default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666 [id=914642e9e7974554a76a94e205db108f]) successfully deleted
I20250115 14:51:58.818248 1167 tablet_replica.cc:331] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.818235 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42] alter complete (version 3)
I20250115 14:51:58.818428 1167 raft_consensus.cc:2238] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.818589 1167 raft_consensus.cc:2267] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:58.819010 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b2c385edf4414157845b971aaf366831 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2c385edf4414157845b971aaf366831 already in progress: creating tablet
I20250115 14:51:58.819103 1167 ts_tablet_manager.cc:1905] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.819453 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 076278b53996407c8a5c73a389459eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.819685 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b60ff80679a470cb4bdced71cc14195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:58.821274 1168 tablet_bootstrap.cc:654] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.821560 1732 raft_consensus.cc:2743] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:58.821775 983 leader_election.cc:343] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.822118 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_49e2a633e2274a808b994184c4d5d28b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:58.822271 1540 raft_consensus.cc:491] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.822415 1540 raft_consensus.cc:513] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:58.822569 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49e2a633e2274a808b994184c4d5d28b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:58.823032 1540 leader_election.cc:290] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:58.823951 710 leader_election.cc:343] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: a6b76cb343fb4456ac07fd3c6e9888de
I20250115 14:51:58.824077 710 leader_election.cc:304] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:58.824479 710 leader_election.cc:343] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b027233cfd314d0ebda472b4588d3f00
I20250115 14:51:58.824585 710 leader_election.cc:304] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:58.824916 1628 raft_consensus.cc:2743] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.825074 1628 raft_consensus.cc:2743] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.825780 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f327ffa731c4c949b24a34dc54ae260" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:58.826398 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac57fa232ae4462793b021b60ede0973" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:58.826615 909 raft_consensus.cc:2463] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:58.826618 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ca710cea154be5b0f57ace4af4168b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:58.827140 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 076278b53996407c8a5c73a389459eaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 076278b53996407c8a5c73a389459eaa already in progress: creating tablet
W20250115 14:51:58.827390 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6b60ff80679a470cb4bdced71cc14195 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b60ff80679a470cb4bdced71cc14195 already in progress: creating tablet
W20250115 14:51:58.834889 984 leader_election.cc:343] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.835147 984 leader_election.cc:304] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.836938 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f327ffa731c4c949b24a34dc54ae260" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.837417 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac57fa232ae4462793b021b60ede0973" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.838032 1698 leader_election.cc:290] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.840760 1167 ts_tablet_manager.cc:1918] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.840889 1167 log.cc:1198] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ab1a38186c5042ff851f57979ab80dbc
I20250115 14:51:58.841149 1168 tablet_bootstrap.cc:492] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.841267 1167 ts_tablet_manager.cc:1939] T ab1a38186c5042ff851f57979ab80dbc P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.841321 1168 ts_tablet_manager.cc:1397] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20250115 14:51:58.841436 1017 tablet_service.cc:1467] Processing CreateTablet for tablet a48227ec5aa846c3ac63bd29ed38be25 (DEFAULT_TABLE table=default.Table_69b643bd59a847959f3f29e23f007498 [id=924019d3b4e842c2b3f173b307885151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:58.842090 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 316) took 1856 ms. Trace:
I20250115 14:51:58.842212 1167 rpcz_store.cc:276] 0115 14:51:56.985494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:56.985532 (+ 38us) service_pool.cc:225] Handling call
0115 14:51:58.818165 (+1832633us) ts_tablet_manager.cc:1145] Deleting tablet ab1a38186c5042ff851f57979ab80dbc
0115 14:51:58.818174 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.837440 (+ 19266us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.840625 (+ 3185us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.842053 (+ 1428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794576}
I20250115 14:51:58.842563 1167 tablet_replica.cc:331] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.842782 1167 raft_consensus.cc:2238] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.842974 1167 raft_consensus.cc:2267] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.843500 1167 ts_tablet_manager.cc:1905] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:58.843914 983 leader_election.cc:343] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.845271 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b60ff80679a470cb4bdced71cc14195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.845619 1168 raft_consensus.cc:357] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.845824 1168 raft_consensus.cc:383] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.845896 1168 raft_consensus.cc:738] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.846107 1168 consensus_queue.cc:260] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.846210 1540 raft_consensus.cc:2743] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.846508 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 076278b53996407c8a5c73a389459eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:58.846730 1168 ts_tablet_manager.cc:1428] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20250115 14:51:58.847013 1168 tablet_bootstrap.cc:492] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:58.850576 1168 tablet_bootstrap.cc:654] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.853292 983 leader_election.cc:304] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:58.858134 1540 raft_consensus.cc:2798] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:58.858244 1540 raft_consensus.cc:491] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:58.858350 1540 raft_consensus.cc:3054] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:58.864166 984 leader_election.cc:343] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.864394 984 leader_election.cc:304] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.864652 1168 tablet_bootstrap.cc:492] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.864809 1168 ts_tablet_manager.cc:1397] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
W20250115 14:51:58.865540 984 leader_election.cc:343] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.865636 1168 raft_consensus.cc:357] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.865854 1168 raft_consensus.cc:383] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.865945 1168 raft_consensus.cc:738] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.866158 1168 consensus_queue.cc:260] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.869087 1168 ts_tablet_manager.cc:1428] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:51:58.869390 1168 tablet_bootstrap.cc:492] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:58.869722 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48227ec5aa846c3ac63bd29ed38be25. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:58.870054 1732 raft_consensus.cc:491] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.870646 1167 ts_tablet_manager.cc:1918] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.870783 1167 log.cc:1198] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b25dce523e724066b203e6111cad8111
I20250115 14:51:58.870695 1732 raft_consensus.cc:513] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.871007 1698 raft_consensus.cc:2743] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.871057 1168 tablet_bootstrap.cc:654] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.871147 1167 ts_tablet_manager.cc:1939] T b25dce523e724066b203e6111cad8111 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.871948 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 323) took 1866 ms. Trace:
I20250115 14:51:58.872068 1167 rpcz_store.cc:276] 0115 14:51:57.004918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.005469 (+ 551us) service_pool.cc:225] Handling call
0115 14:51:58.842498 (+1837029us) ts_tablet_manager.cc:1145] Deleting tablet b25dce523e724066b203e6111cad8111
0115 14:51:58.842508 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.867248 (+ 24740us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.870535 (+ 3287us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.871881 (+ 1346us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1807181}
I20250115 14:51:58.872208 1540 raft_consensus.cc:513] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.872615 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b25dce523e724066b203e6111cad8111 (table default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]) successfully deleted
I20250115 14:51:58.873095 1732 leader_election.cc:290] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.873884 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7309ecb11d844848638c042c9bbe5f1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:58.873981 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7309ecb11d844848638c042c9bbe5f1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.874099 909 raft_consensus.cc:2463] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:58.881608 983 leader_election.cc:304] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:58.884630 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 326) took 1876 ms. Trace:
I20250115 14:51:58.884763 1167 rpcz_store.cc:276] 0115 14:51:57.007723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.009374 (+ 1651us) service_pool.cc:225] Handling call
0115 14:51:58.884412 (+1875038us) ts_tablet_manager.cc:1145] Deleting tablet f847f92cd122423aa93f1afa701b0bbb
0115 14:51:58.884428 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.884556 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1836377}
W20250115 14:51:58.885360 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f847f92cd122423aa93f1afa701b0bbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f847f92cd122423aa93f1afa701b0bbb already in progress: creating tablet
I20250115 14:51:58.885855 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 324) took 1878 ms. Trace:
I20250115 14:51:58.885985 1167 rpcz_store.cc:276] 0115 14:51:57.007502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.009309 (+ 1807us) service_pool.cc:225] Handling call
0115 14:51:58.885681 (+1876372us) ts_tablet_manager.cc:1145] Deleting tablet 3e990a2920b542f69a8bdd022db6e7e2
0115 14:51:58.885693 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.885786 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1833256}
W20250115 14:51:58.886440 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3e990a2920b542f69a8bdd022db6e7e2 because the tablet was not found. No further retry: Not found: Tablet not found: 3e990a2920b542f69a8bdd022db6e7e2
I20250115 14:51:58.886493 1168 tablet_bootstrap.cc:492] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.886649 1168 ts_tablet_manager.cc:1397] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20250115 14:51:58.886821 1167 tablet_replica.cc:331] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.887073 1167 raft_consensus.cc:2238] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.887296 1167 raft_consensus.cc:2267] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.887858 1167 ts_tablet_manager.cc:1905] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.887813 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac57fa232ae4462793b021b60ede0973" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:58.888059 909 raft_consensus.cc:3054] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:58.889328 984 leader_election.cc:343] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.890766 909 raft_consensus.cc:2463] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:58.892352 1168 raft_consensus.cc:357] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.892452 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac57fa232ae4462793b021b60ede0973" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:58.892863 1168 raft_consensus.cc:383] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.892951 1168 raft_consensus.cc:738] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.893438 1624 raft_consensus.cc:491] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.893584 1624 raft_consensus.cc:513] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.894210 1624 leader_election.cc:290] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:58.895221 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9e2c053d624b6ba71b031a591475a4" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:58.895153 1168 consensus_queue.cc:260] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.895735 1168 ts_tablet_manager.cc:1428] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250115 14:51:58.896027 1168 tablet_bootstrap.cc:492] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:58.896386 1167 ts_tablet_manager.cc:1918] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.896512 1167 log.cc:1198] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d86075ae5b894d5986ac4bd538f4d42a
I20250115 14:51:58.896977 1167 ts_tablet_manager.cc:1939] T d86075ae5b894d5986ac4bd538f4d42a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.897742 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 321) took 1892 ms. Trace:
I20250115 14:51:58.897820 1168 tablet_bootstrap.cc:654] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.897876 1167 rpcz_store.cc:276] 0115 14:51:57.004753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.005429 (+ 676us) service_pool.cc:225] Handling call
0115 14:51:58.886694 (+1881265us) ts_tablet_manager.cc:1145] Deleting tablet d86075ae5b894d5986ac4bd538f4d42a
0115 14:51:58.886728 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.893268 (+ 6540us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.896199 (+ 2931us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.897693 (+ 1494us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1818480}
I20250115 14:51:58.898370 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d86075ae5b894d5986ac4bd538f4d42a (table default.Table_c5f3c652f95240ff9f080165af17f5e6 [id=54e9cab363084972abe02521b2cc19b8]) successfully deleted
W20250115 14:51:58.900595 848 leader_election.cc:343] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.903823 1732 raft_consensus.cc:2798] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:58.903927 1732 raft_consensus.cc:491] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:58.904029 1732 raft_consensus.cc:3054] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:58.906849 1167 tablet_replica.cc:331] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.906843 1732 raft_consensus.cc:513] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.907088 1167 raft_consensus.cc:2238] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:58.907384 1167 pending_rounds.cc:62] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:58.907469 1732 leader_election.cc:290] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.907455 1167 pending_rounds.cc:69] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558325688410112 op_type: NO_OP noop_request { }
I20250115 14:51:58.907562 1167 raft_consensus.cc:2883] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:58.907662 1167 raft_consensus.cc:2267] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.907862 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9e2c053d624b6ba71b031a591475a4" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:58.908231 1167 ts_tablet_manager.cc:1905] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.908772 1049 raft_consensus.cc:2463] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:58.909322 847 leader_election.cc:304] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.909878 1624 raft_consensus.cc:2798] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:58.909986 1624 raft_consensus.cc:491] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:58.910086 1624 raft_consensus.cc:3054] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:58.911991 1168 tablet_bootstrap.cc:492] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.912125 1168 ts_tablet_manager.cc:1397] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250115 14:51:58.914691 1167 ts_tablet_manager.cc:1918] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:58.914812 1167 log.cc:1198] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d4fb2d33f3c745beb4a4ba036e23a8e0
I20250115 14:51:58.915213 1167 ts_tablet_manager.cc:1939] T d4fb2d33f3c745beb4a4ba036e23a8e0 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.915948 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 322) took 1911 ms. Trace:
I20250115 14:51:58.912900 1168 raft_consensus.cc:357] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.916138 1168 raft_consensus.cc:383] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.916257 1168 raft_consensus.cc:738] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.916875 1168 consensus_queue.cc:260] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.917105 1624 raft_consensus.cc:513] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.918076 1624 leader_election.cc:290] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:58.918617 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9e2c053d624b6ba71b031a591475a4" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:58.919236 848 leader_election.cc:343] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.919549 1750 raft_consensus.cc:491] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.919729 1750 raft_consensus.cc:513] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.920142 983 leader_election.cc:304] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:58.920486 1750 leader_election.cc:290] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.920818 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9e2c053d624b6ba71b031a591475a4" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:58.920944 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7309ecb11d844848638c042c9bbe5f1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:58.921113 909 raft_consensus.cc:3054] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:58.921241 1540 leader_election.cc:290] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.922423 1049 raft_consensus.cc:3054] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:58.923732 909 raft_consensus.cc:2463] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
W20250115 14:51:58.925132 710 leader_election.cc:343] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: d243daba84ff4befa0c4e7bac5fe7476
W20250115 14:51:58.925751 848 connection.cc:537] client connection to 127.0.104.61:45653 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250115 14:51:58.925611 984 leader_election.cc:343] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.927428 1168 ts_tablet_manager.cc:1428] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
W20250115 14:51:58.927655 711 connection.cc:537] client connection to 127.0.104.61:45653 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250115 14:51:58.927999 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7309ecb11d844848638c042c9bbe5f1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:58.916085 1167 rpcz_store.cc:276] 0115 14:51:57.004843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.005448 (+ 605us) service_pool.cc:225] Handling call
0115 14:51:58.906772 (+1901324us) ts_tablet_manager.cc:1145] Deleting tablet d4fb2d33f3c745beb4a4ba036e23a8e0
0115 14:51:58.906785 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.911514 (+ 4729us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.914583 (+ 3069us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.915914 (+ 1331us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1837427}
I20250115 14:51:58.928596 1167 tablet_replica.cc:331] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.928836 1167 raft_consensus.cc:2238] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.929013 1167 raft_consensus.cc:2267] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.927731 1168 tablet_bootstrap.cc:492] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:58.929569 1167 ts_tablet_manager.cc:1905] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:58.929857 441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.61:45653 (error 108) [suppressed 2 similar messages]
I20250115 14:51:58.931169 1168 tablet_bootstrap.cc:654] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.936061 1167 ts_tablet_manager.cc:1918] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.936211 1167 log.cc:1198] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/36685a04e7454f6592f6ab83c356f466
I20250115 14:51:58.936609 1167 ts_tablet_manager.cc:1939] T 36685a04e7454f6592f6ab83c356f466 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.941049 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 336) took 1812 ms. Trace:
I20250115 14:51:58.941160 1167 rpcz_store.cc:276] 0115 14:51:57.128480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.128768 (+ 288us) service_pool.cc:225] Handling call
0115 14:51:58.928536 (+1799768us) ts_tablet_manager.cc:1145] Deleting tablet 36685a04e7454f6592f6ab83c356f466
0115 14:51:58.928546 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.933089 (+ 4543us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.935942 (+ 2853us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.940989 (+ 5047us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1798108}
I20250115 14:51:58.942051 1168 tablet_bootstrap.cc:492] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:58.942499 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 337) took 1813 ms. Trace:
I20250115 14:51:58.942606 1167 rpcz_store.cc:276] 0115 14:51:57.128597 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.128791 (+ 194us) service_pool.cc:225] Handling call
0115 14:51:58.941465 (+1812674us) ts_tablet_manager.cc:1145] Deleting tablet 7d07686c40324233a49e4c49aeed73f6
0115 14:51:58.941475 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.941607 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1810910}
I20250115 14:51:58.942997 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 338) took 1814 ms. Trace:
I20250115 14:51:58.943094 1167 rpcz_store.cc:276] 0115 14:51:57.128723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.128809 (+ 86us) service_pool.cc:225] Handling call
0115 14:51:58.942829 (+1814020us) ts_tablet_manager.cc:1145] Deleting tablet e7100e6a51ed4e178e86138f10d5ffc2
0115 14:51:58.942838 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.942937 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806355}
I20250115 14:51:58.943397 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 340) took 1418 ms. Trace:
I20250115 14:51:58.943478 1167 rpcz_store.cc:276] 0115 14:51:57.524746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.524795 (+ 49us) service_pool.cc:225] Handling call
0115 14:51:58.943275 (+1418480us) ts_tablet_manager.cc:1145] Deleting tablet 629873c00550408798b00489ece4ea5f
0115 14:51:58.943283 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.943350 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418258}
I20250115 14:51:58.943769 1167 tablet_replica.cc:331] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.943970 1167 raft_consensus.cc:2238] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.944137 1167 raft_consensus.cc:2267] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.944686 1167 ts_tablet_manager.cc:1905] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.948653 1168 ts_tablet_manager.cc:1397] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250115 14:51:58.949467 1168 raft_consensus.cc:357] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.949610 1540 raft_consensus.cc:2798] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:58.949699 1168 raft_consensus.cc:383] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:58.949788 1168 raft_consensus.cc:738] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:58.950031 1168 consensus_queue.cc:260] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.952189 1049 raft_consensus.cc:2463] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:58.952250 1168 ts_tablet_manager.cc:1428] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:58.952404 983 leader_election.cc:304] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:58.952580 1168 tablet_bootstrap.cc:492] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:51:58.953258 710 leader_election.cc:343] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d243daba84ff4befa0c4e7bac5fe7476
I20250115 14:51:58.953409 710 leader_election.cc:304] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:58.953828 1750 raft_consensus.cc:2743] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.956269 1168 tablet_bootstrap.cc:654] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:58.956763 847 leader_election.cc:304] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:58.956997 1624 raft_consensus.cc:2798] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:58.957131 1624 raft_consensus.cc:695] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:58.957371 1624 consensus_queue.cc:237] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.967183 1167 ts_tablet_manager.cc:1918] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:58.967320 1167 log.cc:1198] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f50edca843e54f4b8e803d4d180b736d
I20250115 14:51:58.967698 1167 ts_tablet_manager.cc:1939] T f50edca843e54f4b8e803d4d180b736d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.967737 1540 raft_consensus.cc:695] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:58.968043 1540 consensus_queue.cc:237] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.968576 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 341) took 1443 ms. Trace:
I20250115 14:51:58.968776 1750 raft_consensus.cc:491] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.968772 1167 rpcz_store.cc:276] 0115 14:51:57.525168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.525204 (+ 36us) service_pool.cc:225] Handling call
0115 14:51:58.943677 (+1418473us) ts_tablet_manager.cc:1145] Deleting tablet f50edca843e54f4b8e803d4d180b736d
0115 14:51:58.943684 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.947778 (+ 4094us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.967021 (+ 19243us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.968531 (+ 1510us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418279}
I20250115 14:51:58.968976 1750 raft_consensus.cc:513] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.969416 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 339) took 1449 ms. Trace:
I20250115 14:51:58.969571 1167 rpcz_store.cc:276] 0115 14:51:57.520258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.531089 (+ 10831us) service_pool.cc:225] Handling call
0115 14:51:58.969259 (+1438170us) ts_tablet_manager.cc:1145] Deleting tablet 37ad2b7350294375bd30fb4b46906491
0115 14:51:58.969270 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.969379 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1437945}
I20250115 14:51:58.969660 1750 leader_election.cc:290] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:58.969908 1663 raft_consensus.cc:491] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.969955 1167 tablet_replica.cc:331] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.970046 1663 raft_consensus.cc:513] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.970184 1167 raft_consensus.cc:2238] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:58.970536 1167 raft_consensus.cc:2267] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.970749 1663 leader_election.cc:290] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:58.971117 1167 ts_tablet_manager.cc:1905] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:58.971601 710 leader_election.cc:343] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 3ae60842a2d4486eb7651c3f710ee1e8
I20250115 14:51:58.975370 1732 raft_consensus.cc:2798] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:51:58.975417 710 leader_election.cc:343] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 3ae60842a2d4486eb7651c3f710ee1e8
I20250115 14:51:58.975519 710 leader_election.cc:304] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:58.976017 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a6ee485127484aa71d021edb39ab43" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:51:58.976544 847 leader_election.cc:343] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:58.976800 1750 raft_consensus.cc:2743] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:58.978531 1167 ts_tablet_manager.cc:1918] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:58.980765 1167 log.cc:1198] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/2711a7d437f44b2fa11504d8e400e360
I20250115 14:51:58.984272 1167 ts_tablet_manager.cc:1939] T 2711a7d437f44b2fa11504d8e400e360 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:58.985597 1732 raft_consensus.cc:695] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:58.985978 1732 consensus_queue.cc:237] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:58.986785 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 351) took 1418 ms. Trace:
I20250115 14:51:58.986932 1167 rpcz_store.cc:276] 0115 14:51:57.568622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.568662 (+ 40us) service_pool.cc:225] Handling call
0115 14:51:58.969884 (+1401222us) ts_tablet_manager.cc:1145] Deleting tablet 2711a7d437f44b2fa11504d8e400e360
0115 14:51:58.969898 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:58.974460 (+ 4562us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.978430 (+ 3970us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:58.986729 (+ 8299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1401002}
I20250115 14:51:58.987365 1167 tablet_replica.cc:331] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:58.987613 1167 raft_consensus.cc:2238] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:58.987826 1167 raft_consensus.cc:2267] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:58.988726 1167 ts_tablet_manager.cc:1905] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:58.995294 1750 raft_consensus.cc:491] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:58.995457 1750 raft_consensus.cc:513] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:58.996101 1750 leader_election.cc:290] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:51:58.997283 710 leader_election.cc:343] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 1e7580b6245b41628874d9dcbc3366e2
W20250115 14:51:58.998188 710 leader_election.cc:343] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 1e7580b6245b41628874d9dcbc3366e2
I20250115 14:51:58.998370 710 leader_election.cc:304] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:58.998678 1750 raft_consensus.cc:2743] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:59.000695 1162 tablet_bootstrap.cc:492] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.005901 1162 ts_tablet_manager.cc:1397] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.371s user 0.041s sys 0.070s
I20250115 14:51:59.006914 1751 tablet_replica.cc:331] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.007174 1751 raft_consensus.cc:2238] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:59.006788 1162 raft_consensus.cc:357] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.007378 1162 raft_consensus.cc:383] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.007531 1162 raft_consensus.cc:738] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.007716 1751 raft_consensus.cc:2267] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:59.007989 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:45653, user_credentials={real_user=slave}} blocked reactor thread for 78152.9us
I20250115 14:51:59.009383 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d4fb2d33f3c745beb4a4ba036e23a8e0 (table default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]) successfully deleted
I20250115 14:51:59.008581 1751 ts_tablet_manager.cc:1905] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.009967 1750 raft_consensus.cc:491] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.010116 1750 raft_consensus.cc:513] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.010757 1750 leader_election.cc:290] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.011679 1750 raft_consensus.cc:491] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.011821 1750 raft_consensus.cc:513] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:59.011997 710 leader_election.cc:343] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: d1fa787e364d4ca6a4b756a8f4beba3b
I20250115 14:51:59.012404 1750 leader_election.cc:290] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.007795 1162 consensus_queue.cc:260] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:59.012820 710 leader_election.cc:343] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d1fa787e364d4ca6a4b756a8f4beba3b
I20250115 14:51:59.012917 710 leader_election.cc:304] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:59.013936 1162 ts_tablet_manager.cc:1428] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250115 14:51:59.014034 710 leader_election.cc:343] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: ab1a38186c5042ff851f57979ab80dbc
I20250115 14:51:59.014236 1162 tablet_bootstrap.cc:492] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.014576 1167 ts_tablet_manager.cc:1918] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:59.014648 710 leader_election.cc:343] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: ab1a38186c5042ff851f57979ab80dbc
I20250115 14:51:59.014775 710 leader_election.cc:304] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:59.015165 1628 raft_consensus.cc:2743] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:59.015328 1628 raft_consensus.cc:2743] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:59.015350 1167 log.cc:1198] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b9ce445d334842c8a6bacad9f693feda
I20250115 14:51:59.015873 1162 tablet_bootstrap.cc:654] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:59.016242 984 leader_election.cc:343] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.016456 1167 ts_tablet_manager.cc:1939] T b9ce445d334842c8a6bacad9f693feda P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.017324 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 36685a04e7454f6592f6ab83c356f466 (table default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]) successfully deleted
W20250115 14:51:59.017611 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7d07686c40324233a49e4c49aeed73f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d07686c40324233a49e4c49aeed73f6 already in progress: creating tablet
W20250115 14:51:59.017853 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e7100e6a51ed4e178e86138f10d5ffc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7100e6a51ed4e178e86138f10d5ffc2 already in progress: creating tablet
W20250115 14:51:59.018045 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 629873c00550408798b00489ece4ea5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 629873c00550408798b00489ece4ea5f already in progress: creating tablet
I20250115 14:51:59.018116 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a6ee485127484aa71d021edb39ab43" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.018245 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f50edca843e54f4b8e803d4d180b736d (table default.Table_490c22b7496a4d7db6bc6955e246363b [id=93b72d2ab0bc4ca2bf675a110a5bf4f2]) successfully deleted
I20250115 14:51:59.018381 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 365) took 1386 ms. Trace:
W20250115 14:51:59.018441 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 37ad2b7350294375bd30fb4b46906491 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37ad2b7350294375bd30fb4b46906491 already in progress: creating tablet
I20250115 14:51:59.018496 1167 rpcz_store.cc:276] 0115 14:51:57.631663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.631719 (+ 56us) service_pool.cc:225] Handling call
0115 14:51:58.987286 (+1355567us) ts_tablet_manager.cc:1145] Deleting tablet b9ce445d334842c8a6bacad9f693feda
0115 14:51:58.987297 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:59.007364 (+ 20067us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:59.014443 (+ 7079us) tablet_metadata.cc:641] Metadata flushed
0115 14:51:59.018343 (+ 3900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348748}
I20250115 14:51:59.018642 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 2711a7d437f44b2fa11504d8e400e360 (table default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8 [id=a02ee91f154e41fd9ee54b476fd77331]) successfully deleted
W20250115 14:51:59.018816 848 leader_election.cc:343] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.018921 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 364) took 1398 ms. Trace:
I20250115 14:51:59.018961 848 leader_election.cc:304] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:59.019044 1167 rpcz_store.cc:276] 0115 14:51:57.631462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.632192 (+ 730us) service_pool.cc:225] Handling call
0115 14:51:59.018758 (+1386566us) ts_tablet_manager.cc:1145] Deleting tablet 4abde784f9c948cb96e6cc93c30817bd
0115 14:51:59.018768 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:59.018886 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1345200}
I20250115 14:51:59.019382 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 375) took 1125 ms. Trace:
I20250115 14:51:59.019500 1663 raft_consensus.cc:2743] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:59.020416 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b9ce445d334842c8a6bacad9f693feda (table default.Table_ff3a0751d563485db14fc5a44116f2ac [id=4334f62c5f0a43119d21690c19315404]) successfully deleted
W20250115 14:51:59.020650 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 4abde784f9c948cb96e6cc93c30817bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4abde784f9c948cb96e6cc93c30817bd already in progress: creating tablet
I20250115 14:51:59.019490 1167 rpcz_store.cc:276] 0115 14:51:57.893430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.902147 (+ 8717us) service_pool.cc:225] Handling call
0115 14:51:59.019272 (+1117125us) ts_tablet_manager.cc:1145] Deleting tablet d195be2d543f4a8693045a024912c43a
0115 14:51:59.019280 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:59.019351 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107551}
I20250115 14:51:59.021318 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 376) took 1118 ms. Trace:
I20250115 14:51:59.021476 1167 rpcz_store.cc:276] 0115 14:51:57.902552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.903165 (+ 613us) service_pool.cc:225] Handling call
0115 14:51:59.021193 (+1118028us) ts_tablet_manager.cc:1145] Deleting tablet 07a000ab73dd46ac8e134784a156040a
0115 14:51:59.021201 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:59.021284 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1104419}
I20250115 14:51:59.021879 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 377) took 1113 ms. Trace:
I20250115 14:51:59.022038 1167 rpcz_store.cc:276] 0115 14:51:57.908761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.911742 (+ 2981us) service_pool.cc:225] Handling call
0115 14:51:59.021748 (+1110006us) ts_tablet_manager.cc:1145] Deleting tablet caa1827366a34f039fd12deb855f07b6
0115 14:51:59.021756 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:59.021838 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101901}
I20250115 14:51:59.022472 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 379) took 1102 ms. Trace:
I20250115 14:51:59.022647 1167 rpcz_store.cc:276] 0115 14:51:57.919653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.921563 (+ 1910us) service_pool.cc:225] Handling call
0115 14:51:59.022339 (+1100776us) ts_tablet_manager.cc:1145] Deleting tablet d71a1ecd900942e5ae788c9a28c70a7a
0115 14:51:59.022347 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:59.022437 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100585}
I20250115 14:51:59.023029 1732 raft_consensus.cc:491] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.023226 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 374) took 1137 ms. Trace:
I20250115 14:51:59.023168 1732 raft_consensus.cc:513] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:59.023617 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d195be2d543f4a8693045a024912c43a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d195be2d543f4a8693045a024912c43a already in progress: creating tablet
W20250115 14:51:59.024011 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 07a000ab73dd46ac8e134784a156040a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a000ab73dd46ac8e134784a156040a already in progress: creating tablet
I20250115 14:51:59.024016 1732 leader_election.cc:290] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.024503 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe696560ba6b4a76ad013d9a3486e398" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:59.024732 911 raft_consensus.cc:2463] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:59.024780 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe696560ba6b4a76ad013d9a3486e398" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.025074 983 leader_election.cc:304] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:59.025334 1698 raft_consensus.cc:2798] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:59.025418 1698 raft_consensus.cc:491] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:59.025513 1698 raft_consensus.cc:3054] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:59.026945 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet caa1827366a34f039fd12deb855f07b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caa1827366a34f039fd12deb855f07b6 already in progress: creating tablet
W20250115 14:51:59.027344 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d71a1ecd900942e5ae788c9a28c70a7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d71a1ecd900942e5ae788c9a28c70a7a already in progress: creating tablet
I20250115 14:51:59.028050 1698 raft_consensus.cc:513] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.028965 1698 leader_election.cc:290] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.029248 1653 tablet.cc:1721] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:59.029222 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe696560ba6b4a76ad013d9a3486e398" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:59.029450 911 raft_consensus.cc:3054] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.032225 911 raft_consensus.cc:2463] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:59.033300 983 leader_election.cc:304] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:59.023372 1167 rpcz_store.cc:276] 0115 14:51:57.885713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.885754 (+ 41us) service_pool.cc:225] Handling call
0115 14:51:59.023013 (+1137259us) ts_tablet_manager.cc:1145] Deleting tablet 90c1602aef514dd29d5d2cffb043dbb6
0115 14:51:59.023022 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:59.023098 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1099257}
I20250115 14:51:59.033600 1540 raft_consensus.cc:2798] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:59.033728 1540 raft_consensus.cc:695] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
W20250115 14:51:59.033890 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 90c1602aef514dd29d5d2cffb043dbb6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90c1602aef514dd29d5d2cffb043dbb6 already in progress: creating tablet
I20250115 14:51:59.034102 1540 consensus_queue.cc:237] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.034266 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 378) took 1116 ms. Trace:
I20250115 14:51:59.034467 1167 rpcz_store.cc:276] 0115 14:51:57.917816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.918005 (+ 189us) service_pool.cc:225] Handling call
0115 14:51:59.033616 (+1115611us) ts_tablet_manager.cc:1145] Deleting tablet f8c81f47cd6e4a308cd62124db38d42f
0115 14:51:59.033627 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:59.034230 (+ 603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106092}
W20250115 14:51:59.038851 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f8c81f47cd6e4a308cd62124db38d42f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8c81f47cd6e4a308cd62124db38d42f already in progress: creating tablet
I20250115 14:51:59.039259 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 380) took 1106 ms. Trace:
I20250115 14:51:59.039413 1167 rpcz_store.cc:276] 0115 14:51:57.930608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:57.931418 (+ 810us) service_pool.cc:225] Handling call
0115 14:51:59.037018 (+1105600us) ts_tablet_manager.cc:1145] Deleting tablet e198b4e1a65b4704bf2bb8f7d5987385
0115 14:51:59.037027 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:51:59.037115 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1096602}
I20250115 14:51:59.041121 1731 tablet.cc:1721] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:59.041785 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe696560ba6b4a76ad013d9a3486e398" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:51:59.041896 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e198b4e1a65b4704bf2bb8f7d5987385 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e198b4e1a65b4704bf2bb8f7d5987385 already in progress: creating tablet
W20250115 14:51:59.042273 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d8c53fdf16243f0a614e48ec4ac94f4 already in progress: creating tablet
W20250115 14:51:59.042757 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 75788f8eb36b446e927197f624fbfdf5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75788f8eb36b446e927197f624fbfdf5 already in progress: creating tablet
I20250115 14:51:59.043151 1167 tablet_replica.cc:331] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:59.043403 1167 raft_consensus.cc:2238] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.043641 1167 raft_consensus.cc:2267] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.044314 1167 ts_tablet_manager.cc:1905] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.049201 1652 tablet.cc:1721] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250115 14:51:59.049358 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_b8b42fde86054733ac70ea83d43d6f1b [id=be99ec07535243a68c1b3ee1be0c62e7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 consensus state has no leader
I20250115 14:51:59.050128 1751 ts_tablet_manager.cc:1918] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:59.050256 1751 log.cc:1198] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6b60ff80679a470cb4bdced71cc14195
I20250115 14:51:59.050652 1751 ts_tablet_manager.cc:1939] T 6b60ff80679a470cb4bdced71cc14195 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.051594 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6b60ff80679a470cb4bdced71cc14195 (table default.Table_c088519ba27c457983f341363f8a109e [id=201c214e2eab4873af2095e83dc88b23]) successfully deleted
W20250115 14:51:59.052063 984 leader_election.cc:343] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.052656 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0e924821d12045f6bc658461675aead4
W20250115 14:51:59.053910 984 leader_election.cc:343] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:59.054075 441 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b -> Peer 7d97cf5f1fbb4bc6883711020e2fa3f3 (127.0.104.61:45653): Couldn't send request to peer 7d97cf5f1fbb4bc6883711020e2fa3f3. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45653: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:59.054651 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e924821d12045f6bc658461675aead4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.056030 451 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet fe696560ba6b4a76ad013d9a3486e398 (table default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]). Expected version 1 got 0
I20250115 14:51:59.058609 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 50273a0019bb4cd8977f7ef014372d85
I20250115 14:51:59.058974 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6f07ef30cac2470d9b2f428338dd37a3
I20250115 14:51:59.059731 449 master_service.cc:946] ReplaceTablet: received request to replace tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.062117 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f07ef30cac2470d9b2f428338dd37a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.062333 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 50273a0019bb4cd8977f7ef014372d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 70425c25919b4d0b999450210e1cb33f) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.053473 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e924821d12045f6bc658461675aead4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:51:59.062682 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0e924821d12045f6bc658461675aead4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e924821d12045f6bc658461675aead4 already in progress: creating tablet
I20250115 14:51:59.062723 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d70c898088424b64902d7ba0be41017d" } new_table_name: "default.Table_ec25afcaa18b41b3906954d80092245d" modify_external_catalogs: true
I20250115 14:51:59.063278 1162 tablet_bootstrap.cc:492] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.063419 1162 ts_tablet_manager.cc:1397] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.001s
I20250115 14:51:59.063414 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 50273a0019bb4cd8977f7ef014372d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 70425c25919b4d0b999450210e1cb33f) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.063603 455 catalog_manager.cc:6596] ReplaceTablet: tablet 50273a0019bb4cd8977f7ef014372d85 deleted and replaced by tablet 70425c25919b4d0b999450210e1cb33f
I20250115 14:51:59.063824 455 master_service.cc:946] ReplaceTablet: received request to replace tablet e3fbde011216426cb41c51fe359dcb5f from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.064322 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 51b5bc2c71cf41809d7057acd47e27be from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.064399 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f07ef30cac2470d9b2f428338dd37a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:59.066977 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 009ddfe9dfe14ba2b067f907a9d13538 consensus state has no leader
I20250115 14:51:59.067219 450 master_service.cc:946] ReplaceTablet: received request to replace tablet e4161fce81834cd2ba4ffb789f6a5ba3 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.067865 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e] alter complete (version 2)
W20250115 14:51:59.068833 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:55879 (request call id 54) took 820 ms (client timeout 1000 ms). Trace:
I20250115 14:51:59.069278 1162 raft_consensus.cc:357] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.069514 1162 raft_consensus.cc:383] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.069591 1162 raft_consensus.cc:738] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.069797 1162 consensus_queue.cc:260] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:51:59.068941 446 rpcz_store.cc:269] 0115 14:51:58.248516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:58.627519 (+379003us) service_pool.cc:225] Handling call
0115 14:51:59.068780 (+441261us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0115 14:51:58.653965 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0115 14:51:58.654065 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0115 14:51:58.654073 (+ 8us) write_op.cc:435] Acquired schema lock
0115 14:51:58.654076 (+ 3us) tablet.cc:659] Decoding operations
0115 14:51:58.654176 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0115 14:51:58.654187 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0115 14:51:58.654192 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0115 14:51:58.654250 (+ 58us) tablet.cc:698] Row locks acquired
0115 14:51:58.654253 (+ 3us) write_op.cc:260] PREPARE: finished
0115 14:51:58.654288 (+ 35us) write_op.cc:270] Start()
0115 14:51:58.654323 (+ 35us) write_op.cc:276] Timestamp: P: 1736952718654259 usec, L: 0
0115 14:51:58.654327 (+ 4us) op_driver.cc:348] REPLICATION: starting
0115 14:51:58.654532 (+ 205us) log.cc:844] Serialized 356 byte log entry
0115 14:51:58.812114 (+157582us) op_driver.cc:464] REPLICATION: finished
0115 14:51:58.813064 (+ 950us) write_op.cc:301] APPLY: starting
0115 14:51:58.813072 (+ 8us) tablet.cc:1364] starting BulkCheckPresence
0115 14:51:58.813095 (+ 23us) tablet.cc:1367] finished BulkCheckPresence
0115 14:51:58.813120 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0115 14:51:58.813148 (+ 28us) tablet.cc:1380] finished ApplyRowOperation cycle
0115 14:51:58.813169 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0115 14:51:58.813176 (+ 7us) write_op.cc:312] APPLY: finished
0115 14:51:58.813220 (+ 44us) log.cc:844] Serialized 23 byte log entry
0115 14:51:58.813253 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
0115 14:51:58.813263 (+ 10us) write_op.cc:454] Released schema lock
0115 14:51:58.813267 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0115 14:51:58.813323 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0115 14:51:58.813327 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0115 14:51:58.819343 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0115 14:51:58.819430 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0115 14:51:58.819434 (+ 4us) write_op.cc:435] Acquired schema lock
0115 14:51:58.819436 (+ 2us) tablet.cc:659] Decoding operations
0115 14:51:58.819510 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0115 14:51:58.819518 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0115 14:51:58.819522 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0115 14:51:58.819565 (+ 43us) tablet.cc:698] Row locks acquired
0115 14:51:58.819568 (+ 3us) write_op.cc:260] PREPARE: finished
0115 14:51:58.819595 (+ 27us) write_op.cc:270] Start()
0115 14:51:58.819620 (+ 25us) write_op.cc:276] Timestamp: P: 1736952718819552 usec, L: 0
0115 14:51:58.819623 (+ 3us) op_driver.cc:348] REPLICATION: starting
0115 14:51:58.819794 (+ 171us) log.cc:844] Serialized 356 byte log entry
0115 14:51:59.064954 (+245160us) op_driver.cc:464] REPLICATION: finished
0115 14:51:59.065975 (+ 1021us) write_op.cc:301] APPLY: starting
0115 14:51:59.065985 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0115 14:51:59.066018 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0115 14:51:59.066042 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0115 14:51:59.066080 (+ 38us) tablet.cc:1380] finished ApplyRowOperation cycle
0115 14:51:59.066103 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0115 14:51:59.066110 (+ 7us) write_op.cc:312] APPLY: finished
0115 14:51:59.066163 (+ 53us) log.cc:844] Serialized 23 byte log entry
0115 14:51:59.066199 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0115 14:51:59.066211 (+ 12us) write_op.cc:454] Released schema lock
0115 14:51:59.066216 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0115 14:51:59.066276 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0115 14:51:59.066279 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":923,"apply.run_cpu_time_us":252,"apply.run_wall_time_us":251,"num_ops":1,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":646,"replication_time_us":157773}],["op",{"apply.queue_time_us":989,"apply.run_cpu_time_us":296,"apply.run_wall_time_us":296,"num_ops":1,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":501,"prepare.run_wall_time_us":500,"replication_time_us":245315}]]}
I20250115 14:51:59.071830 1162 ts_tablet_manager.cc:1428] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:51:59.072129 1162 tablet_bootstrap.cc:492] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.073889 1162 tablet_bootstrap.cc:654] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.074400 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet f50edca843e54f4b8e803d4d180b736d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.075416 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9e2c053d624b6ba71b031a591475a4 with delete_type TABLET_DATA_DELETED (replaced by tablet d5ffa997ca6c46559830b1dedf2d1a66) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.075589 1164 tablet_bootstrap.cc:492] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.075754 1164 ts_tablet_manager.cc:1397] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.288s user 0.043s sys 0.039s
I20250115 14:51:59.076571 1164 raft_consensus.cc:357] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.076769 1164 raft_consensus.cc:383] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.076848 1164 raft_consensus.cc:738] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.077087 1164 consensus_queue.cc:260] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.077596 1164 ts_tablet_manager.cc:1428] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:59.077908 1164 tablet_bootstrap.cc:492] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.079655 1164 tablet_bootstrap.cc:654] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:59.082083 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 50273a0019bb4cd8977f7ef014372d85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50273a0019bb4cd8977f7ef014372d85 already in progress: creating tablet
W20250115 14:51:59.083221 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 076278b53996407c8a5c73a389459eaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 076278b53996407c8a5c73a389459eaa already in progress: creating tablet
W20250115 14:51:59.084025 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6f07ef30cac2470d9b2f428338dd37a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f07ef30cac2470d9b2f428338dd37a3 already in progress: creating tablet
W20250115 14:51:59.085100 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 50273a0019bb4cd8977f7ef014372d85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50273a0019bb4cd8977f7ef014372d85 already in progress: creating tablet
W20250115 14:51:59.084689 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6f07ef30cac2470d9b2f428338dd37a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f07ef30cac2470d9b2f428338dd37a3 already in progress: creating tablet
W20250115 14:51:59.085969 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0e924821d12045f6bc658461675aead4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e924821d12045f6bc658461675aead4 already in progress: creating tablet
I20250115 14:51:59.087481 1751 tablet_replica.cc:331] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.087572 751 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b76cb343fb4456ac07fd3c6e9888de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.087868 1166 tablet_replica.cc:331] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.088090 1166 raft_consensus.cc:2238] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.088271 1166 raft_consensus.cc:2267] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.088913 1166 ts_tablet_manager.cc:1905] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.089252 1751 raft_consensus.cc:2238] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:59.089494 1751 pending_rounds.cc:62] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:59.089581 1751 pending_rounds.cc:69] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558335405768704 op_type: NO_OP noop_request { }
I20250115 14:51:59.089749 1751 raft_consensus.cc:2883] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:59.089856 1751 raft_consensus.cc:2267] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.092550 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cdbc0e16d4224a709c7c5aaf69918e64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.093135 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdbc0e16d4224a709c7c5aaf69918e64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.095350 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e924821d12045f6bc658461675aead4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.095458 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b64c63a30a1c4f328986efc44022083b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.095994 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b64c63a30a1c4f328986efc44022083b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.096616 1168 tablet_bootstrap.cc:492] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:59.096781 1168 ts_tablet_manager.cc:1397] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.144s user 0.002s sys 0.001s
I20250115 14:51:59.097649 1168 raft_consensus.cc:357] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.097872 1168 raft_consensus.cc:383] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.097981 1168 raft_consensus.cc:738] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.098229 1168 consensus_queue.cc:260] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.098775 1168 ts_tablet_manager.cc:1428] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:59.098734 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0f7932fd4ac249c18e0f79b0acb45cc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:59.099050 1168 tablet_bootstrap.cc:492] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:51:59.099195 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f7932fd4ac249c18e0f79b0acb45cc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.099640 1167 ts_tablet_manager.cc:1918] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:59.099802 1167 log.cc:1198] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f960c88df14242c5ada1ecf9d8e68196
I20250115 14:51:59.100237 1167 ts_tablet_manager.cc:1939] T f960c88df14242c5ada1ecf9d8e68196 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.101387 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ca74aa81918e4ee08ddb4801b0cbff16" } new_table_name: "default.Table_7b4753acbcca4cdc94831798b3d47baa" modify_external_catalogs: true
I20250115 14:51:59.102458 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f07ef30cac2470d9b2f428338dd37a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.102600 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 50273a0019bb4cd8977f7ef014372d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 70425c25919b4d0b999450210e1cb33f) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.103533 1162 tablet_bootstrap.cc:492] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.104614 1168 tablet_bootstrap.cc:654] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.104840 1751 ts_tablet_manager.cc:1905] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.105551 1162 ts_tablet_manager.cc:1397] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250115 14:51:59.106362 1162 raft_consensus.cc:357] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.106596 1162 raft_consensus.cc:383] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.106675 1162 raft_consensus.cc:738] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.106842 1166 ts_tablet_manager.cc:1918] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.107036 1166 log.cc:1198] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/a6b76cb343fb4456ac07fd3c6e9888de
I20250115 14:51:59.107422 1166 ts_tablet_manager.cc:1939] T a6b76cb343fb4456ac07fd3c6e9888de P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.108943 1162 consensus_queue.cc:260] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.111176 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f960c88df14242c5ada1ecf9d8e68196 (table default.Table_56a365aae805419683d1f4dfd97951cc [id=24d4e203bce64fcdbd5b6fa30a99e6b2]) successfully deleted
I20250115 14:51:59.111230 1162 ts_tablet_manager.cc:1428] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:51:59.111526 1162 tablet_bootstrap.cc:492] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:59.111515 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b2a11b7b0dd6494daf7b017edd4317a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2a11b7b0dd6494daf7b017edd4317a5 already in progress: creating tablet
W20250115 14:51:59.112335 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 8e6cc08065494d9e85eeff5044b44ba6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e6cc08065494d9e85eeff5044b44ba6 already in progress: creating tablet
I20250115 14:51:59.113059 1167 tablet_replica.cc:331] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:59.113260 1167 raft_consensus.cc:2238] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.113437 1167 raft_consensus.cc:2267] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.113973 1167 ts_tablet_manager.cc:1905] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.117270 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ae60842a2d4486eb7651c3f710ee1e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.117604 1166 tablet_replica.cc:331] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.117859 1166 raft_consensus.cc:2238] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.118085 1166 raft_consensus.cc:2267] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.118651 1166 ts_tablet_manager.cc:1905] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.124673 1162 tablet_bootstrap.cc:654] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.125123 1164 tablet_bootstrap.cc:492] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.125295 1164 ts_tablet_manager.cc:1397] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.007s
I20250115 14:51:59.125424 751 tablet_service.cc:1514] Processing DeleteTablet for tablet d1fa787e364d4ca6a4b756a8f4beba3b with delete_type TABLET_DATA_DELETED (replaced by tablet 79aad920dd9744d9b87a8234c92232d8) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.126093 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b027233cfd314d0ebda472b4588d3f00 with delete_type TABLET_DATA_DELETED (replaced by tablet 9145bbaf3e234f079e355454f3ce28cb) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.126097 1164 raft_consensus.cc:357] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.126344 1164 raft_consensus.cc:383] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.126413 1164 raft_consensus.cc:738] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.126794 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ab1a38186c5042ff851f57979ab80dbc with delete_type TABLET_DATA_DELETED (replaced by tablet 07a000ab73dd46ac8e134784a156040a) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.127648 1164 consensus_queue.cc:260] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.128273 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f55c75e5febb407398b0bc06a1db9b1a with delete_type TABLET_DATA_DELETED (replaced by tablet 0e924821d12045f6bc658461675aead4) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.128799 1164 ts_tablet_manager.cc:1428] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:59.129549 1164 tablet_bootstrap.cc:492] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.130108 1540 raft_consensus.cc:491] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.130255 1540 raft_consensus.cc:513] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.130898 1540 leader_election.cc:290] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.131189 1164 tablet_bootstrap.cc:654] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.131601 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47a8acc3bec46b78725dcc3e675ea74" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:59.131839 909 raft_consensus.cc:2463] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:59.132380 983 leader_election.cc:304] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:59.132689 1540 raft_consensus.cc:2798] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:59.132649 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47a8acc3bec46b78725dcc3e675ea74" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.132783 1540 raft_consensus.cc:491] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:59.132884 1540 raft_consensus.cc:3054] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.135390 1540 raft_consensus.cc:513] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.136377 1540 leader_election.cc:290] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.136726 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47a8acc3bec46b78725dcc3e675ea74" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:59.136891 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47a8acc3bec46b78725dcc3e675ea74" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:59.136961 909 raft_consensus.cc:3054] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:51:59.137290 984 leader_election.cc:343] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:59.137599 984 leader_election.cc:343] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.137693 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 745153c106844076bb020f9ed026c97a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:52 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.137900 1751 ts_tablet_manager.cc:1918] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:59.138016 1751 log.cc:1198] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f50edca843e54f4b8e803d4d180b736d
I20250115 14:51:59.138173 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7580b6245b41628874d9dcbc3366e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.138374 1751 ts_tablet_manager.cc:1939] T f50edca843e54f4b8e803d4d180b736d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.138612 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2711a7d437f44b2fa11504d8e400e360 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.139359 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f50edca843e54f4b8e803d4d180b736d (table default.Table_490c22b7496a4d7db6bc6955e246363b [id=93b72d2ab0bc4ca2bf675a110a5bf4f2]) successfully deleted
I20250115 14:51:59.139688 909 raft_consensus.cc:2463] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:51:59.140074 983 leader_election.cc:304] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:59.140497 1540 raft_consensus.cc:2798] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:59.140925 1540 raft_consensus.cc:695] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:59.141206 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet a6b76cb343fb4456ac07fd3c6e9888de (table default.Table_91244a7b0fa14291bcc2e9a2d776b4ae [id=4a31d55ea43e4ce5bbda5b78fdbfffa6]) successfully deleted
I20250115 14:51:59.141203 1540 consensus_queue.cc:237] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.143409 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet fc3d2ca7cd7c48a79ed1396fbbdc6866
I20250115 14:51:59.143862 449 catalog_manager.cc:6596] ReplaceTablet: tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 deleted and replaced by tablet cd8586f571634e678e783e32d34f887f
I20250115 14:51:59.143868 1164 tablet_bootstrap.cc:492] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.144011 1164 ts_tablet_manager.cc:1397] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
W20250115 14:51:59.144572 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4341977f363d40eeac71b7d33d1b43ac consensus state has no leader
I20250115 14:51:59.144809 1164 raft_consensus.cc:357] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.145030 1164 raft_consensus.cc:383] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.145112 1164 raft_consensus.cc:738] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.145323 1164 consensus_queue.cc:260] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.148955 1164 ts_tablet_manager.cc:1428] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:51:59.149256 1164 tablet_bootstrap.cc:492] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.150663 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 with delete_type TABLET_DATA_DELETED (replaced by tablet cd8586f571634e678e783e32d34f887f) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.150929 1164 tablet_bootstrap.cc:654] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.152860 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4821224ac8f1472d8b1e8afc06a118a3 (table default.Table_721928c380b840ad84f9f5030b52632f [id=c5f831177088497dbf1fee130e231e37]). Expected version 1 got 0
I20250115 14:51:59.154680 1751 tablet_replica.cc:331] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.155459 1751 raft_consensus.cc:2238] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.155709 1751 raft_consensus.cc:2267] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.158509 1751 ts_tablet_manager.cc:1905] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.164666 887 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 with delete_type TABLET_DATA_DELETED (replaced by tablet cd8586f571634e678e783e32d34f887f) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.164754 776 raft_consensus.cc:3054] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.169739 751 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 with delete_type TABLET_DATA_DELETED (replaced by tablet cd8586f571634e678e783e32d34f887f) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.171169 1166 ts_tablet_manager.cc:1918] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.171485 1167 ts_tablet_manager.cc:1918] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.171696 1167 log.cc:1198] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/a73bd0141ccc4ec4b052b18b9fb62bb7
I20250115 14:51:59.172151 1167 ts_tablet_manager.cc:1939] T a73bd0141ccc4ec4b052b18b9fb62bb7 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.172331 1166 log.cc:1198] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/3ae60842a2d4486eb7651c3f710ee1e8
I20250115 14:51:59.172766 1166 ts_tablet_manager.cc:1939] T 3ae60842a2d4486eb7651c3f710ee1e8 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.173887 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet a73bd0141ccc4ec4b052b18b9fb62bb7 (table default.Table_44a23f5fdfb446a59f551a2a7a4ca2ce [id=9a5faae475a948528f47d259a547189b]) successfully deleted
I20250115 14:51:59.175166 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 3ae60842a2d4486eb7651c3f710ee1e8 (table default.Table_edeb6add07054dcba5df71224ae8675c [id=ad886fc755c2448c93964215e2fd5903]) successfully deleted
W20250115 14:51:59.176083 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 076278b53996407c8a5c73a389459eaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 076278b53996407c8a5c73a389459eaa already in progress: creating tablet
I20250115 14:51:59.176465 1166 tablet_replica.cc:331] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.176700 1167 tablet_replica.cc:331] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:59.176967 1166 raft_consensus.cc:2238] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.177057 1168 tablet_bootstrap.cc:492] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:59.177256 1167 raft_consensus.cc:2238] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.177253 1162 tablet_bootstrap.cc:492] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.177502 1166 raft_consensus.cc:2267] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.177500 1162 ts_tablet_manager.cc:1397] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.001s
I20250115 14:51:59.177753 1167 raft_consensus.cc:2267] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.179883 1166 ts_tablet_manager.cc:1905] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.177544 1624 raft_consensus.cc:491] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.180133 1624 raft_consensus.cc:513] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.180887 1624 leader_election.cc:290] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.180601 1162 raft_consensus.cc:357] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.181949 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f327ffa731c4c949b24a34dc54ae260" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.181979 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f327ffa731c4c949b24a34dc54ae260" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:59.182209 1048 raft_consensus.cc:2463] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
W20250115 14:51:59.182526 848 leader_election.cc:343] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.177301 1168 ts_tablet_manager.cc:1397] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.078s user 0.017s sys 0.004s
I20250115 14:51:59.183010 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_00dac00a165544449ce9b8dff8efab98" } new_table_name: "default.Table_bccecf4164694596afcaf1710196a41c" modify_external_catalogs: true
I20250115 14:51:59.183409 1168 raft_consensus.cc:357] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.183606 1168 raft_consensus.cc:383] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.183667 1168 raft_consensus.cc:738] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.183895 1168 consensus_queue.cc:260] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.184896 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_233462856fc84da29a45b15193b81652"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.185457 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_233462856fc84da29a45b15193b81652 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.186394 1168 ts_tablet_manager.cc:1428] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:51:59.186818 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6ccf0c2af49e4f2daf7ada5c0243d0ca" } new_table_name: "default.Table_e1110e8f11bb4423bcdd8fedb129e390" modify_external_catalogs: true
I20250115 14:51:59.187681 1167 ts_tablet_manager.cc:1905] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:59.188051 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 171643c40c9b43ffb823f7c33789284d consensus state has no leader
I20250115 14:51:59.188493 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_eb0e928b8265482990bb5fbf1072985f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.188997 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb0e928b8265482990bb5fbf1072985f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.192356 1168 tablet_bootstrap.cc:492] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:59.192790 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e4161fce81834cd2ba4ffb789f6a5ba3
I20250115 14:51:59.193272 450 catalog_manager.cc:6596] ReplaceTablet: tablet e4161fce81834cd2ba4ffb789f6a5ba3 deleted and replaced by tablet b76d1bbf90c6416ca60805c9d8053bd4
I20250115 14:51:59.193265 1731 tablet.cc:1721] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:59.193696 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c62ee465977947a19aba3314a3e2a3f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.194195 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c62ee465977947a19aba3314a3e2a3f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.196733 1168 tablet_bootstrap.cc:654] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.196861 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet e4161fce81834cd2ba4ffb789f6a5ba3 with delete_type TABLET_DATA_DELETED (replaced by tablet b76d1bbf90c6416ca60805c9d8053bd4) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.199200 447 catalog_manager.cc:5526] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:59.199637 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet fe696560ba6b4a76ad013d9a3486e398 (table default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]). Expected version 1 got 0
I20250115 14:51:59.199777 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e4161fce81834cd2ba4ffb789f6a5ba3 with delete_type TABLET_DATA_DELETED (replaced by tablet b76d1bbf90c6416ca60805c9d8053bd4) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.199990 447 catalog_manager.cc:5526] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:59.200418 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_3afc89f290d048999c9654eeed8bb849 [id=855145809143431d81790dd36147a77b]). Expected version 1 got 0
I20250115 14:51:59.200841 447 catalog_manager.cc:5526] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:59.201200 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:51:59.201895 847 leader_election.cc:304] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:59.202355 1624 raft_consensus.cc:2798] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:59.202491 1624 raft_consensus.cc:491] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:59.202572 1624 raft_consensus.cc:3054] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.205910 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1298fdd0217e4df6bfbafc49b2f2c65c (DEFAULT_TABLE table=default.Table_15773744107143d6b1e5215c832f0a3f [id=f4305d1bfe12463fa68fb1405be5267f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.206329 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1298fdd0217e4df6bfbafc49b2f2c65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.206674 1167 ts_tablet_manager.cc:1918] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:51:59.206805 1167 log.cc:1198] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/6b60ff80679a470cb4bdced71cc14195
I20250115 14:51:59.207152 1167 ts_tablet_manager.cc:1939] T 6b60ff80679a470cb4bdced71cc14195 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.208022 1167 tablet_replica.cc:331] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:59.208201 1167 raft_consensus.cc:2238] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.209254 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_ebc003502c644be98ae8fec069e7bb39 [id=6d0e848982e245e1870c661f7dc35138] alter complete (version 2)
I20250115 14:51:59.210587 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d1b15c846f174bccb711fc94283beb6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.211122 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1b15c846f174bccb711fc94283beb6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.211825 1731 tablet.cc:1716] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: Skipping requested alter to schema version 2, tablet already version 2
I20250115 14:51:59.213050 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_253ac9c181914d0f891baf1f8722dbd0 [id=a724f6e4ad16485b8ee2445bb072be3e] alter complete (version 1)
I20250115 14:51:59.216878 1167 raft_consensus.cc:2267] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.217490 1167 ts_tablet_manager.cc:1905] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.218178 1164 tablet_bootstrap.cc:492] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.218398 1164 ts_tablet_manager.cc:1397] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.009s
I20250115 14:51:59.218531 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a63a0a8bb44fe9be0740fc777e610c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.218719 1751 ts_tablet_manager.cc:1918] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.218763 741 tablet_service.cc:1514] Processing DeleteTablet for tablet d243daba84ff4befa0c4e7bac5fe7476 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.218853 1751 log.cc:1198] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/7a9e2c053d624b6ba71b031a591475a4
I20250115 14:51:59.219189 1751 ts_tablet_manager.cc:1939] T 7a9e2c053d624b6ba71b031a591475a4 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.219261 1164 raft_consensus.cc:357] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.219465 1164 raft_consensus.cc:383] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.219532 1164 raft_consensus.cc:738] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.219794 1164 consensus_queue.cc:260] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.220469 1164 ts_tablet_manager.cc:1428] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:59.222867 1164 tablet_bootstrap.cc:492] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.226418 1164 tablet_bootstrap.cc:654] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.228555 743 tablet_service.cc:1467] Processing CreateTablet for tablet 005a4cd0e9f14e76b01afa49d946ee32 (DEFAULT_TABLE table=default.Table_c0d7ba16034c431d8c39b3ca49d63d88 [id=cfab75a29f6c4e388b90e1814fa72fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.226028 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 09adb34bf9a3464798b9f28c50d6fc11 (DEFAULT_TABLE table=default.Table_a602da76783c41a89e586a035929a1ba [id=37297c9e757d411daac43cbbea813e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.228988 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 005a4cd0e9f14e76b01afa49d946ee32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.229163 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09adb34bf9a3464798b9f28c50d6fc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.226073 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 5df0250d08214a9783c392071126ed4a (DEFAULT_TABLE table=default.Table_82a2ce9b462c48b28d9fa64f0f426ec8 [id=8ffbe34c2a11417f8ca50dbdd627bf22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.232164 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df0250d08214a9783c392071126ed4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.232789 741 tablet_service.cc:1467] Processing CreateTablet for tablet 5df0250d08214a9783c392071126ed4a (DEFAULT_TABLE table=default.Table_82a2ce9b462c48b28d9fa64f0f426ec8 [id=8ffbe34c2a11417f8ca50dbdd627bf22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.233134 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df0250d08214a9783c392071126ed4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.226385 1012 tablet_service.cc:1467] Processing CreateTablet for tablet a2fe51b51994491ab38a644bfc55e3ac (DEFAULT_TABLE table=default.Table_9f7b9d2bd72741fbb3132278c8a77d1c [id=15f2a471b89449459d7dbf804b4e922c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.236616 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2fe51b51994491ab38a644bfc55e3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.236721 749 tablet_service.cc:1467] Processing CreateTablet for tablet a2fe51b51994491ab38a644bfc55e3ac (DEFAULT_TABLE table=default.Table_9f7b9d2bd72741fbb3132278c8a77d1c [id=15f2a471b89449459d7dbf804b4e922c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.237030 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2fe51b51994491ab38a644bfc55e3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.226663 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 005a4cd0e9f14e76b01afa49d946ee32 (DEFAULT_TABLE table=default.Table_c0d7ba16034c431d8c39b3ca49d63d88 [id=cfab75a29f6c4e388b90e1814fa72fae]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:59.240685 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba596cfb3ce64718a42a9083975edc8d consensus state has no leader
I20250115 14:51:59.240701 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 005a4cd0e9f14e76b01afa49d946ee32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.242950 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7ec0e7f2e4de4b14bd6cea8489658118" } modify_external_catalogs: true
I20250115 14:51:59.226930 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 6e616d78933e4f84b9ec2acc51450b18 (DEFAULT_TABLE table=default.Table_a584af21c70545b8a2d3690c20422f1f [id=f8c4ca50220a4dbf9b9480213d836e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.243405 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e616d78933e4f84b9ec2acc51450b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.227205 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 1298fdd0217e4df6bfbafc49b2f2c65c (DEFAULT_TABLE table=default.Table_15773744107143d6b1e5215c832f0a3f [id=f4305d1bfe12463fa68fb1405be5267f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.249971 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1298fdd0217e4df6bfbafc49b2f2c65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.254521 882 tablet_service.cc:1467] Processing CreateTablet for tablet 5df0250d08214a9783c392071126ed4a (DEFAULT_TABLE table=default.Table_82a2ce9b462c48b28d9fa64f0f426ec8 [id=8ffbe34c2a11417f8ca50dbdd627bf22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.255342 887 tablet_service.cc:1514] Processing DeleteTablet for tablet e4161fce81834cd2ba4ffb789f6a5ba3 with delete_type TABLET_DATA_DELETED (replaced by tablet b76d1bbf90c6416ca60805c9d8053bd4) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.255856 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6e616d78933e4f84b9ec2acc51450b18 (DEFAULT_TABLE table=default.Table_a584af21c70545b8a2d3690c20422f1f [id=f8c4ca50220a4dbf9b9480213d836e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.256171 884 tablet_service.cc:1467] Processing CreateTablet for tablet a2fe51b51994491ab38a644bfc55e3ac (DEFAULT_TABLE table=default.Table_9f7b9d2bd72741fbb3132278c8a77d1c [id=15f2a471b89449459d7dbf804b4e922c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.256450 885 tablet_service.cc:1467] Processing CreateTablet for tablet 005a4cd0e9f14e76b01afa49d946ee32 (DEFAULT_TABLE table=default.Table_c0d7ba16034c431d8c39b3ca49d63d88 [id=cfab75a29f6c4e388b90e1814fa72fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.256704 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5df0250d08214a9783c392071126ed4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.257076 891 tablet_service.cc:1467] Processing CreateTablet for tablet 09adb34bf9a3464798b9f28c50d6fc11 (DEFAULT_TABLE table=default.Table_a602da76783c41a89e586a035929a1ba [id=37297c9e757d411daac43cbbea813e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.257444 876 tablet_service.cc:1467] Processing CreateTablet for tablet 1298fdd0217e4df6bfbafc49b2f2c65c (DEFAULT_TABLE table=default.Table_15773744107143d6b1e5215c832f0a3f [id=f4305d1bfe12463fa68fb1405be5267f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.257512 1751 tablet_replica.cc:331] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.257705 1751 raft_consensus.cc:2238] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.257933 1751 raft_consensus.cc:2267] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.258482 1751 ts_tablet_manager.cc:1905] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.258536 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 005a4cd0e9f14e76b01afa49d946ee32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.258882 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2fe51b51994491ab38a644bfc55e3ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.260443 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1298fdd0217e4df6bfbafc49b2f2c65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.258930 1782 raft_consensus.cc:491] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.264443 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e616d78933e4f84b9ec2acc51450b18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.264534 1782 raft_consensus.cc:513] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.267732 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09adb34bf9a3464798b9f28c50d6fc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.273969 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a915e0c0a30940ed8a6233f117e2a560" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:59.274315 1782 leader_election.cc:290] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.274573 1167 ts_tablet_manager.cc:1918] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.274909 1167 log.cc:1198] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/50ca710cea154be5b0f57ace4af4168b
I20250115 14:51:59.275514 1167 ts_tablet_manager.cc:1939] T 50ca710cea154be5b0f57ace4af4168b P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.276412 1751 ts_tablet_manager.cc:1918] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:51:59.276585 1751 log.cc:1198] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e4161fce81834cd2ba4ffb789f6a5ba3
I20250115 14:51:59.277056 1751 ts_tablet_manager.cc:1939] T e4161fce81834cd2ba4ffb789f6a5ba3 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.277863 1167 tablet_replica.cc:331] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:59.279296 1624 raft_consensus.cc:513] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.288277 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f327ffa731c4c949b24a34dc54ae260" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:59.288511 1047 raft_consensus.cc:3054] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.291406 1047 raft_consensus.cc:2463] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:59.291985 847 leader_election.cc:304] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:59.292361 1624 leader_election.cc:290] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.292588 1624 raft_consensus.cc:2798] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:59.292846 1624 raft_consensus.cc:695] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:59.293107 1624 consensus_queue.cc:237] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.302156 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_8e6652dab3bc46b7b4f1b66415398217 [id=8941c6c27037475990cf318614854a27] alter complete (version 2)
I20250115 14:51:59.302294 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7ec0e7f2e4de4b14bd6cea8489658118" } modify_external_catalogs: true
I20250115 14:51:59.303998 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af" } modify_external_catalogs: true
I20250115 14:51:59.304186 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af" } modify_external_catalogs: true
W20250115 14:51:59.255219 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa57c1fa96594a40b2551ad503127e4f consensus state has no leader
I20250115 14:51:59.305298 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ae4cccbf33234ae28a199f7926716b28" } new_table_name: "default.Table_6db88245ecc54e498a261791c7cc2103" modify_external_catalogs: true
I20250115 14:51:59.278191 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 7a9e2c053d624b6ba71b031a591475a4 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]) successfully deleted
W20250115 14:51:59.306581 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 already in progress: creating tablet
I20250115 14:51:59.306650 1782 raft_consensus.cc:491] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.306782 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]) successfully deleted
I20250115 14:51:59.306809 1782 raft_consensus.cc:513] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.307129 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 6b60ff80679a470cb4bdced71cc14195 (table default.Table_c088519ba27c457983f341363f8a109e [id=201c214e2eab4873af2095e83dc88b23]) successfully deleted
I20250115 14:51:59.307358 1782 leader_election.cc:290] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.307394 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 50ca710cea154be5b0f57ace4af4168b (table default.Table_d809e208790b4c1a911708b393be6a3f [id=d45a9a753a874c41a842f25450ab10d2]) successfully deleted
W20250115 14:51:59.307588 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0e924821d12045f6bc658461675aead4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e924821d12045f6bc658461675aead4 already in progress: creating tablet
I20250115 14:51:59.182075 1162 raft_consensus.cc:383] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.308280 1162 raft_consensus.cc:738] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.308563 1162 consensus_queue.cc:260] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.308857 1048 raft_consensus.cc:2463] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
W20250115 14:51:59.307771 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6f07ef30cac2470d9b2f428338dd37a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f07ef30cac2470d9b2f428338dd37a3 already in progress: creating tablet
W20250115 14:51:59.309336 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 50273a0019bb4cd8977f7ef014372d85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50273a0019bb4cd8977f7ef014372d85 already in progress: creating tablet
W20250115 14:51:59.309549 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 already in progress: creating tablet
I20250115 14:51:59.309945 847 leader_election.cc:304] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:59.308670 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0291acf45d75421b8b1c6311560e7f0c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:59.284127 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_97c066067f7a49cab2dd5bdade9f6de2" } modify_external_catalogs: true
I20250115 14:51:59.284679 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fc422b5daf4a4770945aab3d34bc5d35" } modify_external_catalogs: true
I20250115 14:51:59.310798 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_97c066067f7a49cab2dd5bdade9f6de2" } modify_external_catalogs: true
I20250115 14:51:59.310909 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fc422b5daf4a4770945aab3d34bc5d35" } modify_external_catalogs: true
W20250115 14:51:59.311655 847 leader_election.cc:343] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.311764 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_702932472ce94b91b4535e0015a38be1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:59.294375 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fc4f396af4f54518b0232f10057dd159"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.312510 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc4f396af4f54518b0232f10057dd159 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.301498 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 51b5bc2c71cf41809d7057acd47e27be
I20250115 14:51:59.313977 455 catalog_manager.cc:6596] ReplaceTablet: tablet 51b5bc2c71cf41809d7057acd47e27be deleted and replaced by tablet a300a76b8f7942dea648a20cbb3d5368
I20250115 14:51:59.314301 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_89373c553f9c407a9e205bfe0d1acf99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:59.314571 1624 raft_consensus.cc:2798] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250115 14:51:59.314783 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89373c553f9c407a9e205bfe0d1acf99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.301904 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 01291d41a2b345b4bfd00e6be0fb0129
I20250115 14:51:59.316805 1624 raft_consensus.cc:491] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:59.316915 1624 raft_consensus.cc:3054] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.317008 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 9e95d09efd2e44cc81cdcbf87ee453fe from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.318902 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9e2c053d624b6ba71b031a591475a4 with delete_type TABLET_DATA_DELETED (replaced by tablet d5ffa997ca6c46559830b1dedf2d1a66) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.319152 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a6ee485127484aa71d021edb39ab43 with delete_type TABLET_DATA_DELETED (replaced by tablet 00adfe53c726428b9ac2fcb47756b3de) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.319267 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:51:59.319362 882 tablet_service.cc:1514] Processing DeleteTablet for tablet ac57fa232ae4462793b021b60ede0973 with delete_type TABLET_DATA_DELETED (replaced by tablet cea16f889c73400a85b9553c1975457e) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.320235 452 catalog_manager.cc:5526] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:51:59.323570 1167 raft_consensus.cc:2238] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:59.324211 1167 raft_consensus.cc:2267] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.324813 1167 ts_tablet_manager.cc:1905] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.325222 1624 raft_consensus.cc:513] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.310497 1162 ts_tablet_manager.cc:1428] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.133s user 0.002s sys 0.000s
I20250115 14:51:59.326257 1162 tablet_bootstrap.cc:492] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.327214 1168 tablet_bootstrap.cc:492] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:59.327347 1168 ts_tablet_manager.cc:1397] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.141s user 0.005s sys 0.013s
I20250115 14:51:59.328137 1168 raft_consensus.cc:357] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.328370 1168 raft_consensus.cc:383] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.328377 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a915e0c0a30940ed8a6233f117e2a560" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:59.328460 1168 raft_consensus.cc:738] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.328567 1047 raft_consensus.cc:3054] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.328706 1168 consensus_queue.cc:260] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.329370 1168 ts_tablet_manager.cc:1428] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:59.329635 1168 tablet_bootstrap.cc:492] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:59.331893 1047 raft_consensus.cc:2463] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:59.332403 847 leader_election.cc:304] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:51:59.312510 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_702932472ce94b91b4535e0015a38be1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.332706 1782 raft_consensus.cc:2798] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:59.332784 1168 tablet_bootstrap.cc:654] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.332816 1782 raft_consensus.cc:695] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:59.333056 1782 consensus_queue.cc:237] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.334466 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b5bc2c71cf41809d7057acd47e27be with delete_type TABLET_DATA_DELETED (replaced by tablet a300a76b8f7942dea648a20cbb3d5368) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.334712 1751 tablet_replica.cc:331] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.334926 1751 raft_consensus.cc:2238] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.335078 1751 raft_consensus.cc:2267] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.335636 1751 ts_tablet_manager.cc:1905] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.339058 1624 leader_election.cc:290] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.340667 1162 tablet_bootstrap.cc:654] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.342480 1751 ts_tablet_manager.cc:1918] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.342625 1751 log.cc:1198] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/51b5bc2c71cf41809d7057acd47e27be
I20250115 14:51:59.343006 1751 ts_tablet_manager.cc:1939] T 51b5bc2c71cf41809d7057acd47e27be P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.345369 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aad920dd9744d9b87a8234c92232d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.345993 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.346571 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 5703fc5995184298bf72f9e43efb41f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e6cc08065494d9e85eeff5044b44ba6) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.347111 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet ac57fa232ae4462793b021b60ede0973 with delete_type TABLET_DATA_DELETED (replaced by tablet cea16f889c73400a85b9553c1975457e) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.347610 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f847f92cd122423aa93f1afa701b0bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 83415f1c07ad495cabd287bdcd54dad4) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.348125 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f50edca843e54f4b8e803d4d180b736d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.352813 1162 tablet_bootstrap.cc:492] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.352996 1162 ts_tablet_manager.cc:1397] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.005s
I20250115 14:51:59.353773 1162 raft_consensus.cc:357] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.354012 1162 raft_consensus.cc:383] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.354074 1162 raft_consensus.cc:738] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.355576 1750 raft_consensus.cc:491] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.355767 1750 raft_consensus.cc:513] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.359141 1162 consensus_queue.cc:260] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.360755 748 tablet_service.cc:1467] Processing CreateTablet for tablet 09adb34bf9a3464798b9f28c50d6fc11 (DEFAULT_TABLE table=default.Table_a602da76783c41a89e586a035929a1ba [id=37297c9e757d411daac43cbbea813e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.361192 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09adb34bf9a3464798b9f28c50d6fc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.361517 1162 ts_tablet_manager.cc:1428] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250115 14:51:59.361858 1162 tablet_bootstrap.cc:492] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.363652 1162 tablet_bootstrap.cc:654] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.367630 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b5bc2c71cf41809d7057acd47e27be with delete_type TABLET_DATA_DELETED (replaced by tablet a300a76b8f7942dea648a20cbb3d5368) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.371146 1750 leader_election.cc:290] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.372221 884 tablet_service.cc:1514] Processing DeleteTablet for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.372862 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 01291d41a2b345b4bfd00e6be0fb0129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.373020 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 377d711f052545ddbe499afe7ba3de28
I20250115 14:51:59.374480 1770 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:51:59.374770 1770 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: This master's current role is: LEADER
I20250115 14:51:59.374886 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9e95d09efd2e44cc81cdcbf87ee453fe
I20250115 14:51:59.375355 447 catalog_manager.cc:6596] ReplaceTablet: tablet 9e95d09efd2e44cc81cdcbf87ee453fe deleted and replaced by tablet 0fd94e4a7fef44f5be462e7a77e8a123
I20250115 14:51:59.375540 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 259ac764ca844df9b6c0ce1688a188da with delete_type TABLET_DATA_DELETED (replaced by tablet bc8337d181d44ac7bb82551a677b6831) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.375612 447 master_service.cc:946] ReplaceTablet: received request to replace tablet ff9b57b8b9434f55ace5a4f47380df54 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.374661 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet cd8586f571634e678e783e32d34f887f
I20250115 14:51:59.378152 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 377d711f052545ddbe499afe7ba3de28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.378194 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b1b29136da42404eb226ca785d5aa7bc" } new_table_name: "default.Table_544d606cf9af4accb670d91d7d717948" modify_external_catalogs: true
I20250115 14:51:59.378708 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 01291d41a2b345b4bfd00e6be0fb0129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.379518 741 tablet_service.cc:1467] Processing CreateTablet for tablet 6e616d78933e4f84b9ec2acc51450b18 (DEFAULT_TABLE table=default.Table_a584af21c70545b8a2d3690c20422f1f [id=f8c4ca50220a4dbf9b9480213d836e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.379959 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e616d78933e4f84b9ec2acc51450b18. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:51:59.381135 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5dd41d42331494e9c1b021816d0287e consensus state has no leader
I20250115 14:51:59.381801 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 742792be08444792892bd9cd7336dda3
I20250115 14:51:59.382956 1162 tablet_bootstrap.cc:492] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.382942 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 01291d41a2b345b4bfd00e6be0fb0129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.383103 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b5bc2c71cf41809d7057acd47e27be with delete_type TABLET_DATA_DELETED (replaced by tablet a300a76b8f7942dea648a20cbb3d5368) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.383123 1162 ts_tablet_manager.cc:1397] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20250115 14:51:59.387188 1162 raft_consensus.cc:357] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.387415 1162 raft_consensus.cc:383] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.387502 1162 raft_consensus.cc:738] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.387727 1162 consensus_queue.cc:260] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.388715 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4239747374aa4d6d866961430786677c" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:59.389230 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4239747374aa4d6d866961430786677c" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:59.391389 1162 ts_tablet_manager.cc:1428] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:51:59.391698 1162 tablet_bootstrap.cc:492] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.394198 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 377d711f052545ddbe499afe7ba3de28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.394306 1750 raft_consensus.cc:491] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.395903 1750 raft_consensus.cc:513] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.396620 1750 leader_election.cc:290] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.398662 1162 tablet_bootstrap.cc:654] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.401772 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a915e0c0a30940ed8a6233f117e2a560" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.402134 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f327ffa731c4c949b24a34dc54ae260" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:59.405195 1162 tablet_bootstrap.cc:492] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.405372 1162 ts_tablet_manager.cc:1397] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20250115 14:51:59.406121 1162 raft_consensus.cc:357] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.406337 1162 raft_consensus.cc:383] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.406396 1162 raft_consensus.cc:738] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.406620 1162 consensus_queue.cc:260] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.407181 1162 ts_tablet_manager.cc:1428] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:59.407534 1162 tablet_bootstrap.cc:492] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.410912 1164 tablet_bootstrap.cc:492] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.411067 1164 ts_tablet_manager.cc:1397] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.188s user 0.014s sys 0.048s
I20250115 14:51:59.411245 1162 tablet_bootstrap.cc:654] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.413846 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e95d09efd2e44cc81cdcbf87ee453fe with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd94e4a7fef44f5be462e7a77e8a123) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.414533 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 51b5bc2c71cf41809d7057acd47e27be (table default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]) successfully deleted
I20250115 14:51:59.416872 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 377d711f052545ddbe499afe7ba3de28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.417235 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e95d09efd2e44cc81cdcbf87ee453fe with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd94e4a7fef44f5be462e7a77e8a123) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:59.418032 848 leader_election.cc:343] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.418361 1164 raft_consensus.cc:357] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.418432 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a915e0c0a30940ed8a6233f117e2a560" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:59.418910 1164 raft_consensus.cc:383] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.418934 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0291acf45d75421b8b1c6311560e7f0c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.419337 1164 raft_consensus.cc:738] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.419593 1164 consensus_queue.cc:260] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.422767 1164 ts_tablet_manager.cc:1428] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
W20250115 14:51:59.423142 710 leader_election.cc:343] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 2711a7d437f44b2fa11504d8e400e360
I20250115 14:51:59.423352 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_55c2a1e317e34c3cabd92d742e6f2d6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:59.423679 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_86cbb9218b1e442f8537cf2dd2d329af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:59.423897 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a26e212086b1478cb159097102934dd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:59.424165 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 742792be08444792892bd9cd7336dda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.424309 710 leader_election.cc:304] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:51:59.424736 710 leader_election.cc:343] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 2711a7d437f44b2fa11504d8e400e360
W20250115 14:51:59.424885 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55c2a1e317e34c3cabd92d742e6f2d6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:59.425451 848 leader_election.cc:343] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.425925 1164 tablet_bootstrap.cc:492] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.426262 1750 raft_consensus.cc:2743] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:59.426420 1751 tablet_replica.cc:331] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.426591 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 742792be08444792892bd9cd7336dda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.426613 1751 raft_consensus.cc:2238] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.426793 1751 raft_consensus.cc:2267] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:59.427019 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86cbb9218b1e442f8537cf2dd2d329af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.427266 776 raft_consensus.cc:3054] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.428017 710 leader_election.cc:304] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:51:59.428395 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f1d3fac40c0e41be85f83948d69ed4bb" } new_table_name: "default.Table_272f9650cb3142f781f1d4d7c4769dfe" modify_external_catalogs: true
I20250115 14:51:59.428655 1162 tablet_bootstrap.cc:492] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.428805 1162 ts_tablet_manager.cc:1397] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.002s
I20250115 14:51:59.429405 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:45653: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45653: connect: Connection refused (error 111),
I20250115 14:51:59.429534 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250115 14:51:59.433313 1750 raft_consensus.cc:2743] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:59.433374 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a26e212086b1478cb159097102934dd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.434018 1751 ts_tablet_manager.cc:1905] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.435547 1164 tablet_bootstrap.cc:654] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.436898 1540 consensus_queue.cc:1035] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:51:59.441119 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:51:59.429553 1162 raft_consensus.cc:357] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.441877 1162 raft_consensus.cc:383] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.442045 1162 raft_consensus.cc:738] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.442266 1162 consensus_queue.cc:260] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.442461 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f47a8acc3bec46b78725dcc3e675ea74 with delete_type TABLET_DATA_DELETED (replaced by tablet ea8ab6a34caa486ab929af2cac3ce957) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.444236 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 742792be08444792892bd9cd7336dda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.445348 1624 raft_consensus.cc:491] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.445453 1162 ts_tablet_manager.cc:1428] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250115 14:51:59.445505 1624 raft_consensus.cc:513] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.445715 1162 tablet_bootstrap.cc:492] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.446171 1624 leader_election.cc:290] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.447300 1162 tablet_bootstrap.cc:654] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:51:59.447321 847 leader_election.cc:343] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 51b5bc2c71cf41809d7057acd47e27be
I20250115 14:51:59.447525 1166 ts_tablet_manager.cc:1918] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.447906 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4239747374aa4d6d866961430786677c (table default.Table_8e6652dab3bc46b7b4f1b66415398217 [id=8941c6c27037475990cf318614854a27]). Expected version 2 got 0
I20250115 14:51:59.448143 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e95d09efd2e44cc81cdcbf87ee453fe with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd94e4a7fef44f5be462e7a77e8a123) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.448740 1624 raft_consensus.cc:491] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.448890 1624 raft_consensus.cc:513] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.449446 1624 leader_election.cc:290] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.449925 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c75a77d7154b5cb05a5d9ac8a20787" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:59.450129 1047 raft_consensus.cc:2463] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:59.450510 847 leader_election.cc:304] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:59.450815 1624 raft_consensus.cc:2798] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:59.450888 1624 raft_consensus.cc:491] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:59.450966 1624 raft_consensus.cc:3054] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.454721 1624 raft_consensus.cc:513] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.455718 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c75a77d7154b5cb05a5d9ac8a20787" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:59.455905 1047 raft_consensus.cc:3054] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.456799 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 0291acf45d75421b8b1c6311560e7f0c with delete_type TABLET_DATA_DELETED (replaced by tablet f90e4edc7aea43268d581b441b788f27) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.457386 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51b5bc2c71cf41809d7057acd47e27be" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.457934 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c75a77d7154b5cb05a5d9ac8a20787" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.458334 1047 raft_consensus.cc:2463] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:59.458372 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80c75a77d7154b5cb05a5d9ac8a20787" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:59.459285 847 leader_election.cc:304] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:59.460171 1624 leader_election.cc:290] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.460387 1624 raft_consensus.cc:2798] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:59.460642 1624 raft_consensus.cc:695] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:59.460932 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aad920dd9744d9b87a8234c92232d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.459748 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8d0846f2d6464fdd98780567261d93f3" } new_table_name: "default.Table_1ff8a32badfe4ea8997fac383c0e5559" modify_external_catalogs: true
I20250115 14:51:59.461316 882 tablet_service.cc:1514] Processing DeleteTablet for tablet f47a8acc3bec46b78725dcc3e675ea74 with delete_type TABLET_DATA_DELETED (replaced by tablet ea8ab6a34caa486ab929af2cac3ce957) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.461644 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5703fc5995184298bf72f9e43efb41f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e6cc08065494d9e85eeff5044b44ba6) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:59.461884 848 leader_election.cc:343] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.461979 848 leader_election.cc:304] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:59.462203 1663 raft_consensus.cc:2743] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:59.462343 848 leader_election.cc:343] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:59.462599 848 leader_election.cc:343] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.462705 848 leader_election.cc:304] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:51:59.463032 848 leader_election.cc:343] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.463117 1663 raft_consensus.cc:2743] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:51:59.463308 848 leader_election.cc:343] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.460935 1624 consensus_queue.cc:237] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.465509 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ff9b57b8b9434f55ace5a4f47380df54
I20250115 14:51:59.465924 447 catalog_manager.cc:6596] ReplaceTablet: tablet ff9b57b8b9434f55ace5a4f47380df54 deleted and replaced by tablet 8f8bae9867d745afb1dfdf47bcb881da
W20250115 14:51:59.466277 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet caed629878594954842dfc6a40cc8b6d consensus state has no leader
I20250115 14:51:59.468747 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9b57b8b9434f55ace5a4f47380df54 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8bae9867d745afb1dfdf47bcb881da) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.469247 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 3b82924e735846b2914520684d1b8fc7 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.469285 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9b57b8b9434f55ace5a4f47380df54 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8bae9867d745afb1dfdf47bcb881da) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.472335 1167 ts_tablet_manager.cc:1918] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:51:59.473062 1166 log.cc:1198] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d1fa787e364d4ca6a4b756a8f4beba3b
I20250115 14:51:59.473232 1540 raft_consensus.cc:491] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.473376 1540 raft_consensus.cc:513] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.473515 1166 ts_tablet_manager.cc:1939] T d1fa787e364d4ca6a4b756a8f4beba3b P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.474007 1540 leader_election.cc:290] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.474447 1166 tablet_replica.cc:331] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.474619 1166 raft_consensus.cc:2238] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.474802 1166 raft_consensus.cc:2267] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.475278 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0291acf45d75421b8b1c6311560e7f0c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.475379 1166 ts_tablet_manager.cc:1905] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:51:59.475838 984 leader_election.cc:343] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.480460 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d1fa787e364d4ca6a4b756a8f4beba3b (table default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]) successfully deleted
I20250115 14:51:59.481496 1164 tablet_bootstrap.cc:492] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.481681 1164 ts_tablet_manager.cc:1397] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.007s
I20250115 14:51:59.482167 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 276abefcab6a422aa6467f00864c01f4 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.482493 1164 raft_consensus.cc:357] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.482739 1164 raft_consensus.cc:383] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.483652 1162 tablet_bootstrap.cc:492] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.483824 1162 ts_tablet_manager.cc:1397] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.001s
I20250115 14:51:59.484861 1731 tablet.cc:1721] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:59.474536 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0291acf45d75421b8b1c6311560e7f0c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:59.485237 911 raft_consensus.cc:2463] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:51:59.485245 1162 raft_consensus.cc:357] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.485435 1162 raft_consensus.cc:383] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.485509 1162 raft_consensus.cc:738] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.485673 983 leader_election.cc:304] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:59.485765 1162 consensus_queue.cc:260] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.486121 1728 raft_consensus.cc:2798] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:59.486207 1728 raft_consensus.cc:491] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250115 14:51:59.486271 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4c984613c7d45d5b7c513cd900c9558 consensus state has no leader
I20250115 14:51:59.486303 1728 raft_consensus.cc:3054] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.486383 1162 ts_tablet_manager.cc:1428] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:51:59.487910 1162 tablet_bootstrap.cc:492] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:59.489118 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b68c168048649c5b91600eb4302b0ce consensus state has no leader
I20250115 14:51:59.488158 1164 raft_consensus.cc:738] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.489463 1164 consensus_queue.cc:260] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.490778 1728 raft_consensus.cc:513] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.491371 1728 leader_election.cc:290] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.491979 1162 tablet_bootstrap.cc:654] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.491917 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0291acf45d75421b8b1c6311560e7f0c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:51:59.492087 911 raft_consensus.cc:3054] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.493304 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0291acf45d75421b8b1c6311560e7f0c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:59.494685 911 raft_consensus.cc:2463] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
W20250115 14:51:59.494949 984 leader_election.cc:343] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.495898 983 leader_election.cc:304] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:59.496316 1698 raft_consensus.cc:2798] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:59.496461 1698 raft_consensus.cc:695] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:51:59.496690 1698 consensus_queue.cc:237] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.483844 1782 raft_consensus.cc:491] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.484508 882 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9b57b8b9434f55ace5a4f47380df54 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8bae9867d745afb1dfdf47bcb881da) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.499281 1782 raft_consensus.cc:513] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.499825 1164 ts_tablet_manager.cc:1428] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20250115 14:51:59.500813 1164 tablet_bootstrap.cc:492] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.501310 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5703fc5995184298bf72f9e43efb41f5" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:51:59.501492 1047 raft_consensus.cc:2463] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:51:59.501909 847 leader_election.cc:304] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:59.502209 1663 raft_consensus.cc:2798] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:51:59.502285 1663 raft_consensus.cc:491] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:51:59.502372 1663 raft_consensus.cc:3054] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.504225 1164 tablet_bootstrap.cc:654] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.504900 1663 raft_consensus.cc:513] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.505486 1663 leader_election.cc:290] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.506330 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5703fc5995184298bf72f9e43efb41f5" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:51:59.506496 1047 raft_consensus.cc:3054] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.508972 1782 leader_election.cc:290] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:51:59.510206 1162 tablet_bootstrap.cc:492] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.510352 1162 ts_tablet_manager.cc:1397] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20250115 14:51:59.511111 1164 tablet_bootstrap.cc:492] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.511090 1162 raft_consensus.cc:357] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.511242 1164 ts_tablet_manager.cc:1397] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20250115 14:51:59.511255 1162 raft_consensus.cc:383] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.511332 1162 raft_consensus.cc:738] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.511540 1162 consensus_queue.cc:260] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.511883 1164 raft_consensus.cc:357] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.512069 1164 raft_consensus.cc:383] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.512095 1162 ts_tablet_manager.cc:1428] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:59.512157 1164 raft_consensus.cc:738] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.512456 1162 tablet_bootstrap.cc:492] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.512379 1164 consensus_queue.cc:260] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.513047 1047 raft_consensus.cc:2463] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:51:59.514190 1162 tablet_bootstrap.cc:654] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.514225 847 leader_election.cc:304] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:51:59.514619 1166 ts_tablet_manager.cc:1918] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.514870 1782 raft_consensus.cc:2798] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:51:59.515043 1782 raft_consensus.cc:695] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:51:59.515388 1782 consensus_queue.cc:237] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.520669 1166 log.cc:1198] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b027233cfd314d0ebda472b4588d3f00
I20250115 14:51:59.520805 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c763759bb01d449cbd6e2f851b303ec5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.521411 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c763759bb01d449cbd6e2f851b303ec5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.519228 1164 ts_tablet_manager.cc:1428] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250115 14:51:59.521885 1166 ts_tablet_manager.cc:1939] T b027233cfd314d0ebda472b4588d3f00 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.523558 1162 tablet_bootstrap.cc:492] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.523803 1164 tablet_bootstrap.cc:492] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.523872 1162 ts_tablet_manager.cc:1397] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20250115 14:51:59.524751 1162 raft_consensus.cc:357] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.524996 1162 raft_consensus.cc:383] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.525061 1162 raft_consensus.cc:738] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.525274 1162 consensus_queue.cc:260] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.525751 1162 ts_tablet_manager.cc:1428] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:59.526149 1162 tablet_bootstrap.cc:492] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.527302 1164 tablet_bootstrap.cc:654] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.527789 1162 tablet_bootstrap.cc:654] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.527911 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_53bd836a66f34fdea6b15e2c3122ad8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:59.528318 1166 tablet_replica.cc:331] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
W20250115 14:51:59.528478 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53bd836a66f34fdea6b15e2c3122ad8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.528509 1166 raft_consensus.cc:2238] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.528640 1167 log.cc:1198] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e4161fce81834cd2ba4ffb789f6a5ba3
I20250115 14:51:59.528657 1166 raft_consensus.cc:2267] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.529045 1167 ts_tablet_manager.cc:1939] T e4161fce81834cd2ba4ffb789f6a5ba3 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.529178 1166 ts_tablet_manager.cc:1905] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.529469 1167 tablet_replica.cc:331] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:59.529644 1167 raft_consensus.cc:2238] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:59.530033 1167 pending_rounds.cc:62] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:51:59.531090 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_33f4705d7a1b474d82734fe4a80c0424"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.531579 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33f4705d7a1b474d82734fe4a80c0424 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.537529 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_83a9122be22e48a5ad8b22b98c838001" } new_table_name: "default.Table_8b9b25dd8a85434a83d2641ffbc5189e" modify_external_catalogs: true
I20250115 14:51:59.539512 1751 ts_tablet_manager.cc:1918] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.539656 1166 ts_tablet_manager.cc:1918] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.539733 1751 log.cc:1198] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/79aad920dd9744d9b87a8234c92232d8
I20250115 14:51:59.540174 1751 ts_tablet_manager.cc:1939] T 79aad920dd9744d9b87a8234c92232d8 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.540251 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_03c9410818db4ce190e6ca259f472d70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.540802 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03c9410818db4ce190e6ca259f472d70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.541301 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 79aad920dd9744d9b87a8234c92232d8 (table default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]) successfully deleted
I20250115 14:51:59.541669 1162 tablet_bootstrap.cc:492] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.541802 1162 ts_tablet_manager.cc:1397] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20250115 14:51:59.542670 1751 tablet_replica.cc:331] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.543231 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_124bdb85e4194ff5a214a6b79870a1cf" } modify_external_catalogs: true
I20250115 14:51:59.543381 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_124bdb85e4194ff5a214a6b79870a1cf" } modify_external_catalogs: true
I20250115 14:51:59.544476 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]) successfully deleted
I20250115 14:51:59.544502 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b027233cfd314d0ebda472b4588d3f00 (table default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]) successfully deleted
I20250115 14:51:59.540802 1164 tablet_bootstrap.cc:492] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.543571 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5703fc5995184298bf72f9e43efb41f5" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.543846 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5703fc5995184298bf72f9e43efb41f5" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:51:59.548943 1164 ts_tablet_manager.cc:1397] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250115 14:51:59.549145 1167 pending_rounds.cc:69] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558336852172800 op_type: NO_OP noop_request { }
I20250115 14:51:59.549355 1167 raft_consensus.cc:2883] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:59.549499 1167 raft_consensus.cc:2267] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.549854 1164 raft_consensus.cc:357] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.550088 1164 raft_consensus.cc:383] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.550176 1164 raft_consensus.cc:738] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.550407 1164 consensus_queue.cc:260] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.552083 1164 ts_tablet_manager.cc:1428] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:51:59.552413 1164 tablet_bootstrap.cc:492] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.553943 1164 tablet_bootstrap.cc:654] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.556042 1166 log.cc:1198] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ab1a38186c5042ff851f57979ab80dbc
I20250115 14:51:59.556408 1166 ts_tablet_manager.cc:1939] T ab1a38186c5042ff851f57979ab80dbc P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.546308 1168 tablet_bootstrap.cc:492] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:59.558565 1168 ts_tablet_manager.cc:1397] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.229s user 0.034s sys 0.040s
I20250115 14:51:59.559307 1168 raft_consensus.cc:357] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.559513 1168 raft_consensus.cc:383] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.559579 1168 raft_consensus.cc:738] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.559800 1168 consensus_queue.cc:260] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.560134 1166 tablet_replica.cc:331] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.560328 1168 ts_tablet_manager.cc:1428] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:51:59.560328 1166 raft_consensus.cc:2238] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.560488 1166 raft_consensus.cc:2267] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.560603 1168 tablet_bootstrap.cc:492] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:59.560977 1166 ts_tablet_manager.cc:1905] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.562158 1168 tablet_bootstrap.cc:654] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.563220 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ab1a38186c5042ff851f57979ab80dbc (table default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]) successfully deleted
I20250115 14:51:59.567101 1167 ts_tablet_manager.cc:1905] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.569968 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4821224ac8f1472d8b1e8afc06a118a3 (table default.Table_721928c380b840ad84f9f5030b52632f [id=c5f831177088497dbf1fee130e231e37]). Expected version 1 got 0
I20250115 14:51:59.570475 1164 tablet_bootstrap.cc:492] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.570602 1164 ts_tablet_manager.cc:1397] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
W20250115 14:51:59.571517 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3a7a487a42949bd9641c38d6c1fcfd3 consensus state has no leader
I20250115 14:51:59.571354 1164 raft_consensus.cc:357] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.571609 1164 raft_consensus.cc:383] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.571689 1164 raft_consensus.cc:738] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.571930 1164 consensus_queue.cc:260] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:51:59.573251 848 leader_election.cc:343] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:59.573570 848 leader_election.cc:343] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.574242 1164 ts_tablet_manager.cc:1428] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:51:59.574530 1164 tablet_bootstrap.cc:492] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.577244 1164 tablet_bootstrap.cc:654] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.542829 1162 raft_consensus.cc:357] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.579324 1162 raft_consensus.cc:383] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.579447 1162 raft_consensus.cc:738] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.579658 1162 consensus_queue.cc:260] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.580273 1162 ts_tablet_manager.cc:1428] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.038s user 0.000s sys 0.002s
I20250115 14:51:59.580576 1162 tablet_bootstrap.cc:492] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.580689 1751 raft_consensus.cc:2238] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.580852 1751 raft_consensus.cc:2267] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.581389 1751 ts_tablet_manager.cc:1905] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.588562 1166 ts_tablet_manager.cc:1918] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.588956 1166 log.cc:1198] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f55c75e5febb407398b0bc06a1db9b1a
I20250115 14:51:59.589039 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:51:59.589237 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:51:59.589540 1166 ts_tablet_manager.cc:1939] T f55c75e5febb407398b0bc06a1db9b1a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.590634 1166 tablet_replica.cc:331] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.590936 1166 raft_consensus.cc:2238] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.591080 449 catalog_manager.cc:5526] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:51:59.591333 1166 raft_consensus.cc:2267] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.591995 1166 ts_tablet_manager.cc:1905] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.593127 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:51:59.593346 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:51:59.594691 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:51:59.596743 882 tablet_service.cc:1467] Processing CreateTablet for tablet ad988e25800f4af9a069b5a80582f749 (DEFAULT_TABLE table=default.Table_0f7932fd4ac249c18e0f79b0acb45cc2 [id=e3887aa962874cfaa7ee89503ca1a49a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.597163 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad988e25800f4af9a069b5a80582f749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.599900 1166 ts_tablet_manager.cc:1918] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.604005 882 tablet_service.cc:1467] Processing CreateTablet for tablet 248cadcdef8b4e99923cdbca39690685 (DEFAULT_TABLE table=default.Table_b64c63a30a1c4f328986efc44022083b [id=5d393887a5e54536a4139752211d21a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.604279 1166 log.cc:1198] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/745153c106844076bb020f9ed026c97a
I20250115 14:51:59.604383 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 248cadcdef8b4e99923cdbca39690685. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.604861 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 86f0144904ae4008945ccf325039dac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.606640 741 tablet_service.cc:1467] Processing CreateTablet for tablet ad988e25800f4af9a069b5a80582f749 (DEFAULT_TABLE table=default.Table_0f7932fd4ac249c18e0f79b0acb45cc2 [id=e3887aa962874cfaa7ee89503ca1a49a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.607002 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad988e25800f4af9a069b5a80582f749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.606642 1166 ts_tablet_manager.cc:1939] T 745153c106844076bb020f9ed026c97a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.608772 884 tablet_service.cc:1467] Processing CreateTablet for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb (DEFAULT_TABLE table=default.Table_2061d08104dd455788c1d0e8c792abc7 [id=ec452771986240f4a7654505a30d3d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.608826 891 tablet_service.cc:1467] Processing CreateTablet for tablet ac4d850b85e948939008e5275d42e3cc (DEFAULT_TABLE table=default.Table_538a75b4e6e54b619c8a011996f4f971 [id=427b1c6d3389420288bb30b1a4cc8b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.609133 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.582216 1162 tablet_bootstrap.cc:654] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.610090 1166 tablet_replica.cc:331] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.610266 1166 raft_consensus.cc:2238] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.610438 1166 raft_consensus.cc:2267] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:51:59.558353 984 consensus_peers.cc:487] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:51:59.610620 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 52313.4us
I20250115 14:51:59.610940 1166 ts_tablet_manager.cc:1905] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.592119 1028 tablet_service.cc:1467] Processing CreateTablet for tablet ad988e25800f4af9a069b5a80582f749 (DEFAULT_TABLE table=default.Table_0f7932fd4ac249c18e0f79b0acb45cc2 [id=e3887aa962874cfaa7ee89503ca1a49a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.611757 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad988e25800f4af9a069b5a80582f749. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.612382 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4d850b85e948939008e5275d42e3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.614851 741 tablet_service.cc:1467] Processing CreateTablet for tablet 248cadcdef8b4e99923cdbca39690685 (DEFAULT_TABLE table=default.Table_b64c63a30a1c4f328986efc44022083b [id=5d393887a5e54536a4139752211d21a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.615260 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 248cadcdef8b4e99923cdbca39690685. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.592906 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 248cadcdef8b4e99923cdbca39690685 (DEFAULT_TABLE table=default.Table_b64c63a30a1c4f328986efc44022083b [id=5d393887a5e54536a4139752211d21a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.616602 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 248cadcdef8b4e99923cdbca39690685. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.619205 748 tablet_service.cc:1467] Processing CreateTablet for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb (DEFAULT_TABLE table=default.Table_2061d08104dd455788c1d0e8c792abc7 [id=ec452771986240f4a7654505a30d3d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.619526 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.594030 1015 tablet_service.cc:1467] Processing CreateTablet for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb (DEFAULT_TABLE table=default.Table_2061d08104dd455788c1d0e8c792abc7 [id=ec452771986240f4a7654505a30d3d9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.620558 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.594252 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a6ee485127484aa71d021edb39ab43 with delete_type TABLET_DATA_DELETED (replaced by tablet 00adfe53c726428b9ac2fcb47756b3de) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.594487 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet e932d5b820f94130967eccce9e1542fa with delete_type TABLET_DATA_DELETED (replaced by tablet fc3d2ca7cd7c48a79ed1396fbbdc6866) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.624697 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1b52fb251b0a48c2a8a4e07d55d22d3f (DEFAULT_TABLE table=default.Table_d6182b9f2b0e4105887762b69f004053 [id=904e587a8da8478a8b6c9134502377d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.594851 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 0291acf45d75421b8b1c6311560e7f0c with delete_type TABLET_DATA_DELETED (replaced by tablet f90e4edc7aea43268d581b441b788f27) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.602080 1019 tablet_service.cc:1467] Processing CreateTablet for tablet b76d1bbf90c6416ca60805c9d8053bd4 (DEFAULT_TABLE table=default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.625039 1166 ts_tablet_manager.cc:1918] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.625150 1166 log.cc:1198] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/1e7580b6245b41628874d9dcbc3366e2
I20250115 14:51:59.625308 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b76d1bbf90c6416ca60805c9d8053bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.602270 1029 tablet_service.cc:1467] Processing CreateTablet for tablet bf648bb656634def8189fb514a07e144 (DEFAULT_TABLE table=default.Table_49e2a633e2274a808b994184c4d5d28b [id=a00115ecf6954f83b3118654f98e629d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.602399 1016 tablet_service.cc:1467] Processing CreateTablet for tablet ac4d850b85e948939008e5275d42e3cc (DEFAULT_TABLE table=default.Table_538a75b4e6e54b619c8a011996f4f971 [id=427b1c6d3389420288bb30b1a4cc8b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.602344 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 1b52fb251b0a48c2a8a4e07d55d22d3f (DEFAULT_TABLE table=default.Table_d6182b9f2b0e4105887762b69f004053 [id=904e587a8da8478a8b6c9134502377d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.602710 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b (DEFAULT_TABLE table=default.Table_880a5165055d48ffba4f0508b2aebfdb [id=174b82e1e06f4073a74a25aad8fc8045]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.627032 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b52fb251b0a48c2a8a4e07d55d22d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.603183 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 5265d5eeac83481ca97801e3475886da (DEFAULT_TABLE table=default.Table_cdbc0e16d4224a709c7c5aaf69918e64 [id=47db9a9013ec4a2cb3cc0cc8f9caba12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.603613 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 70425c25919b4d0b999450210e1cb33f (DEFAULT_TABLE table=default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.603749 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 1140016d11a74a2cadba494c0b319c54 (DEFAULT_TABLE table=default.Table_06fdf72400894754b2da4a05ae517fac [id=fa2c05b8018545958699fafc77f6e832]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.603860 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 325cd3fd8b4d447998822163e575e758 (DEFAULT_TABLE table=default.Table_52bf98f0d5b44013ba19e15f0fc24cce [id=c7768272a8d14ac0aba3f569f8c8f58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.628531 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task__Z8ae/build/asan/bin/kudu
/tmp/dist-test-task__Z8ae/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:45653
--webserver_interface=127.0.104.61
--webserver_port=40195
--builtin_ntp_servers=127.0.104.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250115 14:51:59.629889 756 tablet_service.cc:1467] Processing CreateTablet for tablet b76d1bbf90c6416ca60805c9d8053bd4 (DEFAULT_TABLE table=default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.630218 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b76d1bbf90c6416ca60805c9d8053bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.630623 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4d850b85e948939008e5275d42e3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.630867 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.631274 740 tablet_service.cc:1467] Processing CreateTablet for tablet 5265d5eeac83481ca97801e3475886da (DEFAULT_TABLE table=default.Table_cdbc0e16d4224a709c7c5aaf69918e64 [id=47db9a9013ec4a2cb3cc0cc8f9caba12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.631642 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5265d5eeac83481ca97801e3475886da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.631994 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf648bb656634def8189fb514a07e144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.631875 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f55c75e5febb407398b0bc06a1db9b1a (table default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]) successfully deleted
I20250115 14:51:59.632546 744 tablet_service.cc:1467] Processing CreateTablet for tablet bf648bb656634def8189fb514a07e144 (DEFAULT_TABLE table=default.Table_49e2a633e2274a808b994184c4d5d28b [id=a00115ecf6954f83b3118654f98e629d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.632644 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 745153c106844076bb020f9ed026c97a (table default.Table_031dcfe40eda4ff49338ab79a0cf9844 [id=9cda5b5b1bcf47dfbc5b5a8dcc69b44a]) successfully deleted
I20250115 14:51:59.632689 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325cd3fd8b4d447998822163e575e758. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.632905 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf648bb656634def8189fb514a07e144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.633273 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1140016d11a74a2cadba494c0b319c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.634112 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5265d5eeac83481ca97801e3475886da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.634846 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b52fb251b0a48c2a8a4e07d55d22d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.635406 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70425c25919b4d0b999450210e1cb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.636667 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_461138269c764aceb6cd3baa42a84638" } modify_external_catalogs: true
I20250115 14:51:59.636843 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_461138269c764aceb6cd3baa42a84638" } modify_external_catalogs: true
I20250115 14:51:59.637724 752 tablet_service.cc:1467] Processing CreateTablet for tablet 70425c25919b4d0b999450210e1cb33f (DEFAULT_TABLE table=default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.638058 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70425c25919b4d0b999450210e1cb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.643303 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 0de6f6f014074c3d8775cb5e4121ccb0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.644614 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d1b15c846f174bccb711fc94283beb6d" } new_table_name: "default.Table_e48cf46cf80d4d98a4955f201d993354" modify_external_catalogs: true
I20250115 14:51:59.646350 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b64c63a30a1c4f328986efc44022083b" } modify_external_catalogs: true
I20250115 14:51:59.647568 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0a8e879b253d48d3b02055cb2bd9663e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.648083 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a8e879b253d48d3b02055cb2bd9663e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.649726 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d551e62017774245b9704a679dbd6e6b" } modify_external_catalogs: true
I20250115 14:51:59.625957 748 tablet_service.cc:1467] Processing CreateTablet for tablet ac4d850b85e948939008e5275d42e3cc (DEFAULT_TABLE table=default.Table_538a75b4e6e54b619c8a011996f4f971 [id=427b1c6d3389420288bb30b1a4cc8b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.626870 749 tablet_service.cc:1467] Processing CreateTablet for tablet 1b52fb251b0a48c2a8a4e07d55d22d3f (DEFAULT_TABLE table=default.Table_d6182b9f2b0e4105887762b69f004053 [id=904e587a8da8478a8b6c9134502377d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.628013 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b (DEFAULT_TABLE table=default.Table_880a5165055d48ffba4f0508b2aebfdb [id=174b82e1e06f4073a74a25aad8fc8045]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.628645 743 tablet_service.cc:1467] Processing CreateTablet for tablet 1140016d11a74a2cadba494c0b319c54 (DEFAULT_TABLE table=default.Table_06fdf72400894754b2da4a05ae517fac [id=fa2c05b8018545958699fafc77f6e832]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.629279 750 tablet_service.cc:1467] Processing CreateTablet for tablet 325cd3fd8b4d447998822163e575e758 (DEFAULT_TABLE table=default.Table_52bf98f0d5b44013ba19e15f0fc24cce [id=c7768272a8d14ac0aba3f569f8c8f58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.651060 1166 ts_tablet_manager.cc:1939] T 1e7580b6245b41628874d9dcbc3366e2 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.651077 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b52fb251b0a48c2a8a4e07d55d22d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.651899 1166 tablet_replica.cc:331] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.652113 1166 raft_consensus.cc:2238] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.654446 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1140016d11a74a2cadba494c0b319c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.655210 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.655877 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325cd3fd8b4d447998822163e575e758. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.657192 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f9a5e1c80cc147cf861197fac6765b01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.657691 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9a5e1c80cc147cf861197fac6765b01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.660635 1168 tablet_bootstrap.cc:492] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:59.661183 1168 ts_tablet_manager.cc:1397] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.001s
I20250115 14:51:59.671180 1162 tablet_bootstrap.cc:492] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.673383 884 tablet_service.cc:1467] Processing CreateTablet for tablet 1140016d11a74a2cadba494c0b319c54 (DEFAULT_TABLE table=default.Table_06fdf72400894754b2da4a05ae517fac [id=fa2c05b8018545958699fafc77f6e832]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.673686 885 tablet_service.cc:1467] Processing CreateTablet for tablet b76d1bbf90c6416ca60805c9d8053bd4 (DEFAULT_TABLE table=default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.673794 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1140016d11a74a2cadba494c0b319c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.674062 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b76d1bbf90c6416ca60805c9d8053bd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.674310 890 tablet_service.cc:1467] Processing CreateTablet for tablet 325cd3fd8b4d447998822163e575e758 (DEFAULT_TABLE table=default.Table_52bf98f0d5b44013ba19e15f0fc24cce [id=c7768272a8d14ac0aba3f569f8c8f58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.674649 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 325cd3fd8b4d447998822163e575e758. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.674804 876 tablet_service.cc:1467] Processing CreateTablet for tablet 5265d5eeac83481ca97801e3475886da (DEFAULT_TABLE table=default.Table_cdbc0e16d4224a709c7c5aaf69918e64 [id=47db9a9013ec4a2cb3cc0cc8f9caba12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.675148 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5265d5eeac83481ca97801e3475886da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.664495 1168 raft_consensus.cc:357] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.686884 1168 raft_consensus.cc:383] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.692953 1164 tablet_bootstrap.cc:492] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.693125 1164 ts_tablet_manager.cc:1397] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.000s
I20250115 14:51:59.693902 1164 raft_consensus.cc:357] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.694137 1164 raft_consensus.cc:383] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.694207 1164 raft_consensus.cc:738] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.692721 1168 raft_consensus.cc:738] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.694433 1164 consensus_queue.cc:260] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.695988 1164 ts_tablet_manager.cc:1428] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:51:59.689695 882 tablet_service.cc:1467] Processing CreateTablet for tablet 70425c25919b4d0b999450210e1cb33f (DEFAULT_TABLE table=default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.696334 1164 tablet_bootstrap.cc:492] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.698045 1164 tablet_bootstrap.cc:654] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.701550 1162 ts_tablet_manager.cc:1397] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.091s user 0.003s sys 0.013s
I20250115 14:51:59.703346 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4d850b85e948939008e5275d42e3cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.655148 1166 raft_consensus.cc:2267] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.707049 1166 ts_tablet_manager.cc:1905] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.709554 1162 raft_consensus.cc:357] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.709794 1162 raft_consensus.cc:383] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.709867 1162 raft_consensus.cc:738] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.710129 1162 consensus_queue.cc:260] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.710858 1162 ts_tablet_manager.cc:1428] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250115 14:51:59.711167 1162 tablet_bootstrap.cc:492] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.703251 1168 consensus_queue.cc:260] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.721163 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b64c63a30a1c4f328986efc44022083b" } modify_external_catalogs: true
I20250115 14:51:59.721216 1167 ts_tablet_manager.cc:1918] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:59.721447 1167 log.cc:1198] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/7a9e2c053d624b6ba71b031a591475a4
I20250115 14:51:59.721817 1167 ts_tablet_manager.cc:1939] T 7a9e2c053d624b6ba71b031a591475a4 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.722203 1167 tablet_replica.cc:331] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:59.722400 1698 raft_consensus.cc:491] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.722419 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 7a9e2c053d624b6ba71b031a591475a4 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]) successfully deleted
I20250115 14:51:59.722530 1698 raft_consensus.cc:513] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.722695 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70425c25919b4d0b999450210e1cb33f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.723115 1698 leader_election.cc:290] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.723699 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7a487a42949bd9641c38d6c1fcfd3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:59.723728 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7a487a42949bd9641c38d6c1fcfd3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:59.724377 984 leader_election.cc:343] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.690438 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b (DEFAULT_TABLE table=default.Table_880a5165055d48ffba4f0508b2aebfdb [id=174b82e1e06f4073a74a25aad8fc8045]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:51:59.724658 983 leader_election.cc:343] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.724804 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.724787 983 leader_election.cc:304] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:59.725688 1166 ts_tablet_manager.cc:1918] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.725837 1166 log.cc:1198] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/2711a7d437f44b2fa11504d8e400e360
I20250115 14:51:59.726183 1166 ts_tablet_manager.cc:1939] T 2711a7d437f44b2fa11504d8e400e360 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.727977 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d551e62017774245b9704a679dbd6e6b" } modify_external_catalogs: true
I20250115 14:51:59.729396 1698 raft_consensus.cc:2743] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:59.690804 876 tablet_service.cc:1467] Processing CreateTablet for tablet bf648bb656634def8189fb514a07e144 (DEFAULT_TABLE table=default.Table_49e2a633e2274a808b994184c4d5d28b [id=a00115ecf6954f83b3118654f98e629d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:51:59.731148 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf648bb656634def8189fb514a07e144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:51:59.731194 1162 tablet_bootstrap.cc:654] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.737596 1166 tablet_replica.cc:331] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.737881 1166 raft_consensus.cc:2238] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.742368 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 1e7580b6245b41628874d9dcbc3366e2 (table default.Table_843fb099548148f29e7b622a4ad34b61 [id=085d5f79fc6c4bdb87752ecc55d89445]) successfully deleted
I20250115 14:51:59.743077 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 2711a7d437f44b2fa11504d8e400e360 (table default.Table_3b68d0d6bcf64cbb84c284b8c21c3bb8 [id=a02ee91f154e41fd9ee54b476fd77331]) successfully deleted
I20250115 14:51:59.743333 1751 ts_tablet_manager.cc:1918] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.746268 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0
I20250115 14:51:59.746537 1751 log.cc:1198] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a73bd0141ccc4ec4b052b18b9fb62bb7
I20250115 14:51:59.747279 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.747337 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.747851 1751 ts_tablet_manager.cc:1939] T a73bd0141ccc4ec4b052b18b9fb62bb7 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.748559 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3b82924e735846b2914520684d1b8fc7
I20250115 14:51:59.748900 1751 tablet_replica.cc:331] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.749061 450 catalog_manager.cc:6596] ReplaceTablet: tablet 3b82924e735846b2914520684d1b8fc7 deleted and replaced by tablet b9a5e18b334b4062b1ab9f7076856c60
I20250115 14:51:59.749032 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a73bd0141ccc4ec4b052b18b9fb62bb7 (table default.Table_44a23f5fdfb446a59f551a2a7a4ca2ce [id=9a5faae475a948528f47d259a547189b]) successfully deleted
I20250115 14:51:59.749096 1751 raft_consensus.cc:2238] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.749577 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b82924e735846b2914520684d1b8fc7 with delete_type TABLET_DATA_DELETED (replaced by tablet b9a5e18b334b4062b1ab9f7076856c60) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.751718 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5bc975b044dd439dbd1f8f9a82cce751
I20250115 14:51:59.752431 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_014f98e470c94ba4b3446630153a9e75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.753077 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_014f98e470c94ba4b3446630153a9e75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.755265 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc975b044dd439dbd1f8f9a82cce751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.757237 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:51:59.757460 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]). Expected version 2 got 0
I20250115 14:51:59.757655 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a3a7a487a42949bd9641c38d6c1fcfd3 (table default.Table_8b9b25dd8a85434a83d2641ffbc5189e [id=738170c5571a4d958944ed432428e99d]). Expected version 3 got 0
I20250115 14:51:59.749265 1751 raft_consensus.cc:2267] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.749778 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b82924e735846b2914520684d1b8fc7 with delete_type TABLET_DATA_DELETED (replaced by tablet b9a5e18b334b4062b1ab9f7076856c60) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.755353 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc975b044dd439dbd1f8f9a82cce751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.759541 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 75788f8eb36b446e927197f624fbfdf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.759733 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f07ef30cac2470d9b2f428338dd37a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.761678 1167 raft_consensus.cc:2238] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.761905 1167 raft_consensus.cc:2267] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.762465 1167 ts_tablet_manager.cc:1905] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.765041 1751 ts_tablet_manager.cc:1905] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.768658 1167 ts_tablet_manager.cc:1918] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.768805 1167 log.cc:1198] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/99a6ee485127484aa71d021edb39ab43
I20250115 14:51:59.769212 1167 ts_tablet_manager.cc:1939] T 99a6ee485127484aa71d021edb39ab43 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.770219 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 99a6ee485127484aa71d021edb39ab43 (table default.Table_d3300e90dc3845f78b3c63dfbf681dde [id=cba9cd7a6d434f18b9e621b035a3a6e0]) successfully deleted
I20250115 14:51:59.770555 1167 tablet_replica.cc:331] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:59.770757 1167 raft_consensus.cc:2238] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.770901 1167 raft_consensus.cc:2267] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.771396 1167 ts_tablet_manager.cc:1905] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.776844 1168 ts_tablet_manager.cc:1428] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.115s user 0.000s sys 0.004s
I20250115 14:51:59.777899 1168 tablet_bootstrap.cc:492] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:59.779659 1168 tablet_bootstrap.cc:654] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.780913 776 raft_consensus.cc:3054] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:51:59.782642 1162 tablet_bootstrap.cc:492] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.782845 1162 ts_tablet_manager.cc:1397] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.072s user 0.010s sys 0.033s
I20250115 14:51:59.783654 1162 raft_consensus.cc:357] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.783910 1162 raft_consensus.cc:383] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.784036 1162 raft_consensus.cc:738] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.786233 1162 consensus_queue.cc:260] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.786806 1168 tablet_bootstrap.cc:492] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:59.786983 1168 ts_tablet_manager.cc:1397] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20250115 14:51:59.787853 1168 raft_consensus.cc:357] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.788103 1168 raft_consensus.cc:383] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.788203 1168 raft_consensus.cc:738] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.788445 1168 consensus_queue.cc:260] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.788671 1751 ts_tablet_manager.cc:1918] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.788980 1168 ts_tablet_manager.cc:1428] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:51:59.789144 1751 log.cc:1198] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/5703fc5995184298bf72f9e43efb41f5
I20250115 14:51:59.789495 1751 ts_tablet_manager.cc:1939] T 5703fc5995184298bf72f9e43efb41f5 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.792460 1751 tablet_replica.cc:331] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.792774 1751 raft_consensus.cc:2238] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:59.793287 1751 pending_rounds.cc:62] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:59.793720 1162 ts_tablet_manager.cc:1428] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250115 14:51:59.793888 1166 raft_consensus.cc:2267] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.793565 1751 pending_rounds.cc:69] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558336903671808 op_type: NO_OP noop_request { }
I20250115 14:51:59.794595 1166 ts_tablet_manager.cc:1905] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.796625 1168 tablet_bootstrap.cc:492] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:59.798969 1168 tablet_bootstrap.cc:654] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.799539 1751 raft_consensus.cc:2883] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:59.800724 1751 raft_consensus.cc:2267] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.800787 1162 tablet_bootstrap.cc:492] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.803687 1167 ts_tablet_manager.cc:1918] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.803877 1167 log.cc:1198] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ac57fa232ae4462793b021b60ede0973
I20250115 14:51:59.804268 1167 ts_tablet_manager.cc:1939] T ac57fa232ae4462793b021b60ede0973 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.805253 1751 ts_tablet_manager.cc:1905] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.809082 1731 tablet.cc:1721] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:51:59.809165 1167 tablet_replica.cc:331] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:51:59.809350 1167 raft_consensus.cc:2238] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.811363 1164 tablet_bootstrap.cc:492] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.811559 1164 ts_tablet_manager.cc:1397] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.115s user 0.013s sys 0.024s
I20250115 14:51:59.814020 1166 ts_tablet_manager.cc:1918] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:51:59.814615 1166 log.cc:1198] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e4161fce81834cd2ba4ffb789f6a5ba3
I20250115 14:51:59.815122 1166 ts_tablet_manager.cc:1939] T e4161fce81834cd2ba4ffb789f6a5ba3 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.816406 1167 raft_consensus.cc:2267] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.816962 1167 ts_tablet_manager.cc:1905] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.808805 1162 tablet_bootstrap.cc:654] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.819465 1166 tablet_replica.cc:331] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.819870 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 5703fc5995184298bf72f9e43efb41f5 (table default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]) successfully deleted
I20250115 14:51:59.820308 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet ac57fa232ae4462793b021b60ede0973 (table default.Table_fbd024b67b794f99b66433ffbb89184c [id=b3c701a3bba64b0cbedcf0928b3c873c]) successfully deleted
W20250115 14:51:59.821636 984 consensus_peers.cc:487] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:59.822783 1751 ts_tablet_manager.cc:1918] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:51:59.822732 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_56da29ad1fe24546bc5c6a4a17a774e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:51:59.822896 1751 log.cc:1198] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ac57fa232ae4462793b021b60ede0973
W20250115 14:51:59.823343 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56da29ad1fe24546bc5c6a4a17a774e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.816912 1731 tablet.cc:1721] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:51:59.824402 1166 raft_consensus.cc:2238] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.824584 1166 raft_consensus.cc:2267] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.824651 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3ffdd3389c224ce9a51ab9c816721b4b" } new_table_name: "default.Table_8f53bbc08e5340aab38bb0ce8ef7aa16" modify_external_catalogs: true
I20250115 14:51:59.825147 1166 ts_tablet_manager.cc:1905] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.814394 1164 raft_consensus.cc:357] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.828394 1164 raft_consensus.cc:383] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.828497 1164 raft_consensus.cc:738] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.828740 1164 consensus_queue.cc:260] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.829111 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.829175 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b82924e735846b2914520684d1b8fc7 with delete_type TABLET_DATA_DELETED (replaced by tablet b9a5e18b334b4062b1ab9f7076856c60) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.829348 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc975b044dd439dbd1f8f9a82cce751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.829779 1164 ts_tablet_manager.cc:1428] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250115 14:51:59.830051 1164 tablet_bootstrap.cc:492] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.831805 1164 tablet_bootstrap.cc:654] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.834681 1751 ts_tablet_manager.cc:1939] T ac57fa232ae4462793b021b60ede0973 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.835309 1162 tablet_bootstrap.cc:492] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.835459 1162 ts_tablet_manager.cc:1397] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250115 14:51:59.836174 1162 raft_consensus.cc:357] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.836414 1162 raft_consensus.cc:383] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.836485 1162 raft_consensus.cc:738] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.836745 1162 consensus_queue.cc:260] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.838804 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e4161fce81834cd2ba4ffb789f6a5ba3 (table default.Table_13d7e84841db4e7897fa09136b16e197 [id=0ef6f07cf29e46e69bf2a4449187a4e7]) successfully deleted
I20250115 14:51:59.839978 1162 ts_tablet_manager.cc:1428] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:51:59.841161 1162 tablet_bootstrap.cc:492] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.845487 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ac57fa232ae4462793b021b60ede0973 (table default.Table_fbd024b67b794f99b66433ffbb89184c [id=b3c701a3bba64b0cbedcf0928b3c873c]) successfully deleted
I20250115 14:51:59.846388 1162 tablet_bootstrap.cc:654] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.846565 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9f327ffa731c4c949b24a34dc54ae260
I20250115 14:51:59.847147 1166 ts_tablet_manager.cc:1918] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.847607 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f327ffa731c4c949b24a34dc54ae260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.846627 1751 tablet_replica.cc:331] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.848675 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d4c984613c7d45d5b7c513cd900c9558 (table default.Table_272f9650cb3142f781f1d4d7c4769dfe [id=2fb684a787e4420aa9218b369eba5cf2]). Expected version 1 got 0
I20250115 14:51:59.849802 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet e7100e6a51ed4e178e86138f10d5ffc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6942d102069d4c33b23fa2c0b29f94a6) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.850524 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f327ffa731c4c949b24a34dc54ae260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.850714 1751 raft_consensus.cc:2238] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:51:59.850900 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ded663ac1197454bbde3c4c2d526002d has no consensus state
I20250115 14:51:59.850905 1751 raft_consensus.cc:2267] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.851222 455 master_service.cc:946] ReplaceTablet: received request to replace tablet d97c2f443ab040f9bc1924f45379165e from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.851734 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d76649a3901e45f4826621907456750f" } new_table_name: "default.Table_c5f747dc2f6b4cffb7b791ea6456c5b3" modify_external_catalogs: true
I20250115 14:51:59.852281 1166 log.cc:1198] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/07a63a0a8bb44fe9be0740fc777e610c
I20250115 14:51:59.852679 1166 ts_tablet_manager.cc:1939] T 07a63a0a8bb44fe9be0740fc777e610c P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.853163 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 248cadcdef8b4e99923cdbca39690685
I20250115 14:51:59.853825 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dae457edf44a40a69a40030ae24a26ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.854347 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dae457edf44a40a69a40030ae24a26ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.855801 1166 tablet_replica.cc:331] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.855762 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6b336b36c2e2438fbf8698e281702756" } new_table_name: "default.Table_35aa379f66554bac80e34f11e41062c7" modify_external_catalogs: true
I20250115 14:51:59.855996 1166 raft_consensus.cc:2238] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.856180 1166 raft_consensus.cc:2267] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.856719 1166 ts_tablet_manager.cc:1905] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.857169 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_99a5f2ca224f459dbf6bd4d4ec255457"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.857602 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99a5f2ca224f459dbf6bd4d4ec255457 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.859052 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 4294c49da1e044be993100b3cf5e2cea from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.860062 446 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet b9a5e18b334b4062b1ab9f7076856c60
I20250115 14:51:59.851528 1751 ts_tablet_manager.cc:1905] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.860308 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 6e616d78933e4f84b9ec2acc51450b18 from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.861882 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d1198c17efa04e4ea060b6d89580342a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.862326 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1198c17efa04e4ea060b6d89580342a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.864761 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 248cadcdef8b4e99923cdbca39690685 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.865247 1162 tablet_bootstrap.cc:492] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.865386 1162 ts_tablet_manager.cc:1397] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250115 14:51:59.866099 1162 raft_consensus.cc:357] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.866298 1162 raft_consensus.cc:383] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.866391 1162 raft_consensus.cc:738] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.866559 1698 raft_consensus.cc:491] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.866796 1698 raft_consensus.cc:513] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.866914 1162 consensus_queue.cc:260] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.867499 1698 leader_election.cc:290] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.868543 1162 ts_tablet_manager.cc:1428] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250115 14:51:59.868505 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99a6ee485127484aa71d021edb39ab43" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:51:59.868952 1162 tablet_bootstrap.cc:492] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:51:59.869169 984 leader_election.cc:343] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:51:59.869881 983 leader_election.cc:343] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 99a6ee485127484aa71d021edb39ab43
I20250115 14:51:59.870002 983 leader_election.cc:304] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:59.870357 1698 raft_consensus.cc:2743] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:59.875430 1162 tablet_bootstrap.cc:654] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.876405 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 248cadcdef8b4e99923cdbca39690685 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.878536 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f327ffa731c4c949b24a34dc54ae260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.879103 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 248cadcdef8b4e99923cdbca39690685 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.880158 1162 tablet_bootstrap.cc:492] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.880378 1162 ts_tablet_manager.cc:1397] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250115 14:51:59.880628 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 36685a04e7454f6592f6ab83c356f466 with delete_type TABLET_DATA_DELETED (replaced by tablet f06544890cbf460cbe218690e998eb1d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.880856 750 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fb2d33f3c745beb4a4ba036e23a8e0 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f327ffa731c4c949b24a34dc54ae260) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.880928 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 07a63a0a8bb44fe9be0740fc777e610c (table default.Table_2a2f7c950304406484083d56f0b3177e [id=685971a2443b4b16a9af9f3a57d132c8]) successfully deleted
I20250115 14:51:59.881183 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcfab1e1f0349739eff636fb1b04806 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.866401 1751 ts_tablet_manager.cc:1918] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.881385 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 5027fba8fed34db59b484918256abc45 with delete_type TABLET_DATA_DELETED (replaced by tablet 37ad2b7350294375bd30fb4b46906491) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.881481 1751 log.cc:1198] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f847f92cd122423aa93f1afa701b0bbb
I20250115 14:51:59.881815 1751 ts_tablet_manager.cc:1939] T f847f92cd122423aa93f1afa701b0bbb P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.881737 1162 raft_consensus.cc:357] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.881945 1162 raft_consensus.cc:383] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.882018 1162 raft_consensus.cc:738] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.882268 1162 consensus_queue.cc:260] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.883076 1162 ts_tablet_manager.cc:1428] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:51:59.883351 1751 tablet_replica.cc:331] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.883533 1751 raft_consensus.cc:2238] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:59.884156 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f847f92cd122423aa93f1afa701b0bbb (table default.Table_458a573b4a4b4714950723b426f0d265 [id=1fd54f7dafdc4777bd06982391452889]) successfully deleted
W20250115 14:51:59.884445 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f50edca843e54f4b8e803d4d180b736d because the tablet was not found. No further retry: Not found: Tablet not found: f50edca843e54f4b8e803d4d180b736d
W20250115 14:51:59.884670 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 01291d41a2b345b4bfd00e6be0fb0129 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01291d41a2b345b4bfd00e6be0fb0129 already in progress: creating tablet
W20250115 14:51:59.884924 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 377d711f052545ddbe499afe7ba3de28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377d711f052545ddbe499afe7ba3de28 already in progress: creating tablet
W20250115 14:51:59.885126 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9e95d09efd2e44cc81cdcbf87ee453fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e95d09efd2e44cc81cdcbf87ee453fe already in progress: creating tablet
W20250115 14:51:59.885365 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 742792be08444792892bd9cd7336dda3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 742792be08444792892bd9cd7336dda3 already in progress: creating tablet
I20250115 14:51:59.889375 1167 ts_tablet_manager.cc:1918] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.889514 1167 log.cc:1198] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/51b5bc2c71cf41809d7057acd47e27be
I20250115 14:51:59.889855 1167 ts_tablet_manager.cc:1939] T 51b5bc2c71cf41809d7057acd47e27be P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:51:59.890368 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 51b5bc2c71cf41809d7057acd47e27be (table default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]) successfully deleted
I20250115 14:51:59.890441 1166 ts_tablet_manager.cc:1918] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.890548 1166 log.cc:1198] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d243daba84ff4befa0c4e7bac5fe7476
I20250115 14:51:59.890872 1166 ts_tablet_manager.cc:1939] T d243daba84ff4befa0c4e7bac5fe7476 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.891295 1168 tablet_bootstrap.cc:492] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:59.891434 1168 ts_tablet_manager.cc:1397] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.095s user 0.002s sys 0.002s
I20250115 14:51:59.892374 1168 raft_consensus.cc:357] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.892588 1168 raft_consensus.cc:383] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.892675 1168 raft_consensus.cc:738] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.892905 1168 consensus_queue.cc:260] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.893568 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d243daba84ff4befa0c4e7bac5fe7476 (table default.Table_c06f4c91e37e4b6f95521174e05efb24 [id=9a6ed225ff3143f5bd2fc0f25324d322]) successfully deleted
I20250115 14:51:59.894147 1166 tablet_replica.cc:331] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.894337 1166 raft_consensus.cc:2238] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.894490 1166 raft_consensus.cc:2267] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.893615 1168 ts_tablet_manager.cc:1428] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:59.895025 1166 ts_tablet_manager.cc:1905] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.896065 1168 tablet_bootstrap.cc:492] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:51:59.897661 1751 pending_rounds.cc:62] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:59.897794 1751 pending_rounds.cc:69] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558337610031104 op_type: NO_OP noop_request { }
I20250115 14:51:59.899665 1698 raft_consensus.cc:491] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:51:59.899816 1698 raft_consensus.cc:513] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.900827 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92231509c754e8dbfc564813a76a123" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:51:59.901821 984 leader_election.cc:343] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.901902 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92231509c754e8dbfc564813a76a123" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:51:59.906177 1166 ts_tablet_manager.cc:1918] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:51:59.907176 983 leader_election.cc:343] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:51:59.907303 983 leader_election.cc:304] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:51:59.907243 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_af26e85d3ff84c6296a22baee6440af2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.907795 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af26e85d3ff84c6296a22baee6440af2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.907889 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e990a2920b542f69a8bdd022db6e7e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 4ee8b63cc21145d8993b058d8c6d23c7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.907987 1540 raft_consensus.cc:2743] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:51:59.908207 1166 log.cc:1198] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/259ac764ca844df9b6c0ce1688a188da
W20250115 14:51:59.908551 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ca128bf3d6fc430298da932a78f54184 consensus state has no leader
I20250115 14:51:59.908622 1166 ts_tablet_manager.cc:1939] T 259ac764ca844df9b6c0ce1688a188da P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.908973 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_df2dd54e8d8749aa942872c4b20ef257" } new_table_name: "default.Table_1cc96e57499c4c8ba91f9724a57dadbd" modify_external_catalogs: true
I20250115 14:51:59.909366 1698 leader_election.cc:290] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:51:59.909646 1698 consensus_queue.cc:1035] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:51:59.910157 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 010e6535b6bc44f391e6f2b6f1686532 consensus state has no leader
I20250115 14:51:59.911373 1162 tablet_bootstrap.cc:492] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:51:59.911399 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 259ac764ca844df9b6c0ce1688a188da (table default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]) successfully deleted
I20250115 14:51:59.911711 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a7e733ef2d844d238685b7d25b53c151"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.912075 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 01291d41a2b345b4bfd00e6be0fb0129 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01291d41a2b345b4bfd00e6be0fb0129 already in progress: creating tablet
W20250115 14:51:59.912176 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7e733ef2d844d238685b7d25b53c151 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:51:59.913239 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 51b5bc2c71cf41809d7057acd47e27be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51b5bc2c71cf41809d7057acd47e27be already in progress: creating tablet
W20250115 14:51:59.913285 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ce498ffd3f04c5d8eb642739a838903 consensus state has no leader
I20250115 14:51:59.913619 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2c94d0bb9dd54c5195b1fcbf7cec15fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:51:59.913914 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 377d711f052545ddbe499afe7ba3de28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377d711f052545ddbe499afe7ba3de28 already in progress: creating tablet
W20250115 14:51:59.914095 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c94d0bb9dd54c5195b1fcbf7cec15fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:51:59.914222 1162 tablet_bootstrap.cc:654] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.915345 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4294c49da1e044be993100b3cf5e2cea
W20250115 14:51:59.915621 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 742792be08444792892bd9cd7336dda3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 742792be08444792892bd9cd7336dda3 already in progress: creating tablet
I20250115 14:51:59.915752 447 catalog_manager.cc:6596] ReplaceTablet: tablet 4294c49da1e044be993100b3cf5e2cea deleted and replaced by tablet d1259256b7744e739581125611abbe67
I20250115 14:51:59.916194 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4294c49da1e044be993100b3cf5e2cea with delete_type TABLET_DATA_DELETED (replaced by tablet d1259256b7744e739581125611abbe67) from {username='slave'} at 127.0.0.1:46776
W20250115 14:51:59.917033 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9e95d09efd2e44cc81cdcbf87ee453fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e95d09efd2e44cc81cdcbf87ee453fe already in progress: creating tablet
W20250115 14:51:59.917635 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ff9b57b8b9434f55ace5a4f47380df54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff9b57b8b9434f55ace5a4f47380df54 already in progress: creating tablet
I20250115 14:51:59.917933 1166 tablet_replica.cc:331] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.918108 1166 raft_consensus.cc:2238] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.918273 1166 raft_consensus.cc:2267] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.919027 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6e616d78933e4f84b9ec2acc51450b18
I20250115 14:51:59.919459 446 catalog_manager.cc:6596] ReplaceTablet: tablet 6e616d78933e4f84b9ec2acc51450b18 deleted and replaced by tablet ba33eb3f5a40450db5cf115fbe8a17e7
I20250115 14:51:59.919600 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e616d78933e4f84b9ec2acc51450b18 with delete_type TABLET_DATA_DELETED (replaced by tablet ba33eb3f5a40450db5cf115fbe8a17e7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.920013 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 43605c995df4400e91139199c9514bfd from {username='slave'} at 127.0.0.1:47464
I20250115 14:51:59.920089 1166 ts_tablet_manager.cc:1905] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.923043 1751 raft_consensus.cc:2883] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:59.923452 1751 raft_consensus.cc:2267] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.924889 1751 ts_tablet_manager.cc:1905] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.925675 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet b92231509c754e8dbfc564813a76a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.925911 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4102b1f83e414d93a6a68315e36d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.926110 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4294c49da1e044be993100b3cf5e2cea with delete_type TABLET_DATA_DELETED (replaced by tablet d1259256b7744e739581125611abbe67) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.926286 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e616d78933e4f84b9ec2acc51450b18 with delete_type TABLET_DATA_DELETED (replaced by tablet ba33eb3f5a40450db5cf115fbe8a17e7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:51:59.928921 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 4294c49da1e044be993100b3cf5e2cea with delete_type TABLET_DATA_DELETED (replaced by tablet d1259256b7744e739581125611abbe67) from {username='slave'} at 127.0.0.1:56252
I20250115 14:51:59.928979 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e616d78933e4f84b9ec2acc51450b18 with delete_type TABLET_DATA_DELETED (replaced by tablet ba33eb3f5a40450db5cf115fbe8a17e7) from {username='slave'} at 127.0.0.1:56252
W20250115 14:51:59.929425 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92aeca346a7746468dbbff053d933472 consensus state has no leader
I20250115 14:51:59.933064 1166 ts_tablet_manager.cc:1918] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.933249 1166 log.cc:1198] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/86f0144904ae4008945ccf325039dac1
I20250115 14:51:59.933815 1166 ts_tablet_manager.cc:1939] T 86f0144904ae4008945ccf325039dac1 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:51:59.935734 984 consensus_peers.cc:487] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:51:59.935729 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 86f0144904ae4008945ccf325039dac1 (table default.Table_39eb08eb474049f2a2fbe741ef730d68 [id=ab6f49066e884652b77a0b9a791575c7]) successfully deleted
W20250115 14:51:59.936467 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 already in progress: creating tablet
W20250115 14:51:59.937676 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5bc975b044dd439dbd1f8f9a82cce751 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bc975b044dd439dbd1f8f9a82cce751 already in progress: creating tablet
I20250115 14:51:59.938263 1751 ts_tablet_manager.cc:1918] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250115 14:51:59.938315 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9f327ffa731c4c949b24a34dc54ae260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f327ffa731c4c949b24a34dc54ae260 already in progress: creating tablet
I20250115 14:51:59.938807 1751 log.cc:1198] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f47a8acc3bec46b78725dcc3e675ea74
W20250115 14:51:59.938910 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 248cadcdef8b4e99923cdbca39690685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 248cadcdef8b4e99923cdbca39690685 already in progress: creating tablet
I20250115 14:51:59.939189 1166 tablet_replica.cc:331] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.939388 1751 ts_tablet_manager.cc:1939] T f47a8acc3bec46b78725dcc3e675ea74 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.939442 1166 raft_consensus.cc:2238] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.939667 1166 raft_consensus.cc:2267] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.940361 1166 ts_tablet_manager.cc:1905] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.941015 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f47a8acc3bec46b78725dcc3e675ea74 (table default.Table_aa6a106222694bbfbe6c26b0f8e65396 [id=8725c85d30684db495ec7a20962fd75d]) successfully deleted
W20250115 14:51:59.942835 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ff9b57b8b9434f55ace5a4f47380df54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff9b57b8b9434f55ace5a4f47380df54 already in progress: creating tablet
I20250115 14:51:59.943403 1751 tablet_replica.cc:331] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.943822 1751 raft_consensus.cc:2238] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.945176 1751 raft_consensus.cc:2267] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.945937 1751 ts_tablet_manager.cc:1905] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.946367 1166 ts_tablet_manager.cc:1918] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.946561 1166 log.cc:1198] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/36685a04e7454f6592f6ab83c356f466
I20250115 14:51:59.946934 1166 ts_tablet_manager.cc:1939] T 36685a04e7454f6592f6ab83c356f466 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.947921 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 36685a04e7454f6592f6ab83c356f466 (table default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]) successfully deleted
I20250115 14:51:59.948343 1166 tablet_replica.cc:331] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.948576 1166 raft_consensus.cc:2238] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.948838 1166 raft_consensus.cc:2267] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.949599 1166 ts_tablet_manager.cc:1905] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.951051 1164 tablet_bootstrap.cc:492] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.951198 1164 ts_tablet_manager.cc:1397] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.121s user 0.010s sys 0.006s
I20250115 14:51:59.951925 1164 raft_consensus.cc:357] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.952131 1164 raft_consensus.cc:383] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.952190 1164 raft_consensus.cc:738] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.952411 1164 consensus_queue.cc:260] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.952857 1164 ts_tablet_manager.cc:1428] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:51:59.953164 1164 tablet_bootstrap.cc:492] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.954716 1164 tablet_bootstrap.cc:654] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.958334 1751 ts_tablet_manager.cc:1918] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.960036 1751 log.cc:1198] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/99a6ee485127484aa71d021edb39ab43
I20250115 14:51:59.960729 1751 ts_tablet_manager.cc:1939] T 99a6ee485127484aa71d021edb39ab43 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.962128 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 99a6ee485127484aa71d021edb39ab43 (table default.Table_d3300e90dc3845f78b3c63dfbf681dde [id=cba9cd7a6d434f18b9e621b035a3a6e0]) successfully deleted
I20250115 14:51:59.963001 1166 ts_tablet_manager.cc:1918] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.963218 1751 tablet_replica.cc:331] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.963675 1751 raft_consensus.cc:2238] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.964175 1751 raft_consensus.cc:2267] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.965679 1168 tablet_bootstrap.cc:654] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.966126 1751 ts_tablet_manager.cc:1905] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.966578 1164 tablet_bootstrap.cc:492] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.966775 1164 ts_tablet_manager.cc:1397] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20250115 14:51:59.967492 1164 raft_consensus.cc:357] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.967756 1164 raft_consensus.cc:383] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.967835 1164 raft_consensus.cc:738] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.968058 1164 consensus_queue.cc:260] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.969599 1164 ts_tablet_manager.cc:1428] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:51:59.969892 1164 tablet_bootstrap.cc:492] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.972652 1164 tablet_bootstrap.cc:654] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.973415 1166 log.cc:1198] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d4fb2d33f3c745beb4a4ba036e23a8e0
I20250115 14:51:59.973829 1166 ts_tablet_manager.cc:1939] T d4fb2d33f3c745beb4a4ba036e23a8e0 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:51:59.974807 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d4fb2d33f3c745beb4a4ba036e23a8e0 (table default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]) successfully deleted
I20250115 14:51:59.975172 1166 tablet_replica.cc:331] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:51:59.975391 1166 raft_consensus.cc:2238] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:51:59.975594 1166 raft_consensus.cc:2267] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.976330 1166 ts_tablet_manager.cc:1905] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.979012 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d86075ae5b894d5986ac4bd538f4d42a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:51:59.976503 1751 ts_tablet_manager.cc:1918] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.979866 1751 log.cc:1198] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e932d5b820f94130967eccce9e1542fa
I20250115 14:51:59.980638 1751 ts_tablet_manager.cc:1939] T e932d5b820f94130967eccce9e1542fa P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:51:59.981766 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e932d5b820f94130967eccce9e1542fa (table default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]) successfully deleted
I20250115 14:51:59.981737 451 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:51:59.982622 1751 tablet_replica.cc:331] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:51:59.983094 1751 raft_consensus.cc:2238] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:51:59.983647 1751 pending_rounds.cc:62] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:51:59.984009 1751 pending_rounds.cc:69] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558339060846592 op_type: NO_OP noop_request { }
I20250115 14:51:59.984364 1751 raft_consensus.cc:2883] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:51:59.984666 1751 raft_consensus.cc:2267] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:51:59.985953 1751 ts_tablet_manager.cc:1905] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:51:59.986732 1166 ts_tablet_manager.cc:1918] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:51:59.987666 1168 tablet_bootstrap.cc:492] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:51:59.987847 1168 ts_tablet_manager.cc:1397] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.000s
I20250115 14:51:59.988667 1168 raft_consensus.cc:357] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.988914 1168 raft_consensus.cc:383] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.988988 1168 raft_consensus.cc:738] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.989760 1164 tablet_bootstrap.cc:492] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:51:59.990010 1164 ts_tablet_manager.cc:1397] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20250115 14:51:59.990790 1164 raft_consensus.cc:357] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.991092 1164 raft_consensus.cc:383] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.991209 1164 raft_consensus.cc:738] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.991417 1168 consensus_queue.cc:260] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:51:59.991499 1164 consensus_queue.cc:260] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.992449 1168 ts_tablet_manager.cc:1428] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250115 14:51:59.992938 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 because the tablet was not found. No further retry: Not found: Tablet not found: a73bd0141ccc4ec4b052b18b9fb62bb7
I20250115 14:51:59.994058 1162 tablet_bootstrap.cc:492] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:51:59.994297 1162 ts_tablet_manager.cc:1397] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.094s user 0.018s sys 0.013s
I20250115 14:51:59.994557 1164 ts_tablet_manager.cc:1428] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:51:59.994850 1164 tablet_bootstrap.cc:492] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:51:59.995071 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 1839edd0d49b467d8bb31b3001ff89eb (table default.Table_253ac9c181914d0f891baf1f8722dbd0 [id=a724f6e4ad16485b8ee2445bb072be3e]). Expected version 1 got 0
I20250115 14:51:59.995410 1162 raft_consensus.cc:357] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.995926 1162 raft_consensus.cc:383] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:51:59.997018 1162 raft_consensus.cc:738] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:51:59.997309 1162 consensus_queue.cc:260] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:51:59.998739 1164 tablet_bootstrap.cc:654] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:51:59.998782 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 1839edd0d49b467d8bb31b3001ff89eb (table default.Table_253ac9c181914d0f891baf1f8722dbd0 [id=a724f6e4ad16485b8ee2445bb072be3e]). Expected version 1 got 0
I20250115 14:51:59.999728 1166 log.cc:1198] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9bcfab1e1f0349739eff636fb1b04806
I20250115 14:52:00.000154 1166 ts_tablet_manager.cc:1939] T 9bcfab1e1f0349739eff636fb1b04806 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.001470 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9bcfab1e1f0349739eff636fb1b04806 (table default.Table_fa872457db7140b6a13bfc998e32581b [id=4e1c43b0e14844babf0bb0cf8032a3fa]) successfully deleted
I20250115 14:52:00.001869 1166 tablet_replica.cc:331] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.002096 1166 raft_consensus.cc:2238] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.002240 1162 ts_tablet_manager.cc:1428] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:52:00.002321 1166 raft_consensus.cc:2267] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.003046 1166 ts_tablet_manager.cc:1905] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.004880 1162 tablet_bootstrap.cc:492] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:00.006546 1162 tablet_bootstrap.cc:654] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.007314 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 43605c995df4400e91139199c9514bfd
I20250115 14:52:00.007720 450 catalog_manager.cc:6596] ReplaceTablet: tablet 43605c995df4400e91139199c9514bfd deleted and replaced by tablet e910c387e6a24caf95e14b57c8b83e6f
I20250115 14:52:00.008390 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 43605c995df4400e91139199c9514bfd with delete_type TABLET_DATA_DELETED (replaced by tablet e910c387e6a24caf95e14b57c8b83e6f) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.009052 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 43605c995df4400e91139199c9514bfd with delete_type TABLET_DATA_DELETED (replaced by tablet e910c387e6a24caf95e14b57c8b83e6f) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.009344 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 43605c995df4400e91139199c9514bfd with delete_type TABLET_DATA_DELETED (replaced by tablet e910c387e6a24caf95e14b57c8b83e6f) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.010993 452 catalog_manager.cc:5526] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:00.011449 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:52:00.011745 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:52:00.011961 1751 ts_tablet_manager.cc:1918] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:00.012151 1751 log.cc:1198] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/0291acf45d75421b8b1c6311560e7f0c
I20250115 14:52:00.012212 1164 tablet_bootstrap.cc:492] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.012367 1164 ts_tablet_manager.cc:1397] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20250115 14:52:00.012821 1751 ts_tablet_manager.cc:1939] T 0291acf45d75421b8b1c6311560e7f0c P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:00.013236 1164 raft_consensus.cc:357] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.013468 1164 raft_consensus.cc:383] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.013558 1164 raft_consensus.cc:738] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.013797 1164 consensus_queue.cc:260] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.013994 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 0291acf45d75421b8b1c6311560e7f0c (table default.Table_bbe00fc656b44633ae2fd159ae77ffc0 [id=e6f49fcfb7d74c71904b4e6b1bdfee1d]) successfully deleted
I20250115 14:52:00.014901 1751 tablet_replica.cc:331] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:00.015300 1751 raft_consensus.cc:2238] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.015547 1751 raft_consensus.cc:2267] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.016410 1751 ts_tablet_manager.cc:1905] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.016404 749 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4475441e4a467a8d593b742d2a5b80 (DEFAULT_TABLE table=default.Table_c763759bb01d449cbd6e2f851b303ec5 [id=4f4b4f8aa6364221b4a35c423a5631aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.016825 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4475441e4a467a8d593b742d2a5b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.018234 891 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4475441e4a467a8d593b742d2a5b80 (DEFAULT_TABLE table=default.Table_c763759bb01d449cbd6e2f851b303ec5 [id=4f4b4f8aa6364221b4a35c423a5631aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.018601 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4475441e4a467a8d593b742d2a5b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.018695 776 raft_consensus.cc:3054] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:00.024159 1164 ts_tablet_manager.cc:1428] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250115 14:52:00.026417 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aeca9975f1945ad956f47db1ea16a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.026804 1164 tablet_bootstrap.cc:492] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.028144 1782 raft_consensus.cc:491] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.028324 1782 raft_consensus.cc:513] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.028998 1782 leader_election.cc:290] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:00.029989 847 leader_election.cc:343] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 79aad920dd9744d9b87a8234c92232d8
I20250115 14:52:00.030570 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aad920dd9744d9b87a8234c92232d8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:00.032583 848 leader_election.cc:343] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:00.032742 848 leader_election.cc:304] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:00.033197 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 75788f8eb36b446e927197f624fbfdf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.033529 1782 raft_consensus.cc:2743] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:00.033941 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e209cc3424eb4349b5387a32ac96d6bc" } modify_external_catalogs: true
I20250115 14:52:00.034721 1164 tablet_bootstrap.cc:654] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.035502 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d4c984613c7d45d5b7c513cd900c9558 (table default.Table_272f9650cb3142f781f1d4d7c4769dfe [id=2fb684a787e4420aa9218b369eba5cf2]). Expected version 1 got 0
I20250115 14:52:00.035727 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet fe696560ba6b4a76ad013d9a3486e398 (table default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]). Expected version 1 got 0
I20250115 14:52:00.036473 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e209cc3424eb4349b5387a32ac96d6bc" } modify_external_catalogs: true
I20250115 14:52:00.036788 1166 ts_tablet_manager.cc:1918] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.036952 1166 log.cc:1198] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/5027fba8fed34db59b484918256abc45
I20250115 14:52:00.037657 1166 ts_tablet_manager.cc:1939] T 5027fba8fed34db59b484918256abc45 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.038166 1162 tablet_bootstrap.cc:492] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:00.038472 1162 ts_tablet_manager.cc:1397] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.004s
I20250115 14:52:00.038775 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 5027fba8fed34db59b484918256abc45 (table default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]) successfully deleted
I20250115 14:52:00.039122 1166 tablet_replica.cc:331] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.039362 1166 raft_consensus.cc:2238] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.039324 1162 raft_consensus.cc:357] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.039561 1162 raft_consensus.cc:383] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.039613 1166 raft_consensus.cc:2267] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.039655 1162 raft_consensus.cc:738] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.040077 1162 consensus_queue.cc:260] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.040390 1166 ts_tablet_manager.cc:1905] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.045037 1162 ts_tablet_manager.cc:1428] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:52:00.049037 1162 tablet_bootstrap.cc:492] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:00.049412 1164 tablet_bootstrap.cc:492] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.049626 1164 ts_tablet_manager.cc:1397] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20250115 14:52:00.050025 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de6f6f014074c3d8775cb5e4121ccb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.050451 1164 raft_consensus.cc:357] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.050729 1164 raft_consensus.cc:383] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.050855 1164 raft_consensus.cc:738] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.051143 1164 consensus_queue.cc:260] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:00.053543 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3e990a2920b542f69a8bdd022db6e7e2 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3e990a2920b542f69a8bdd022db6e7e2 consensus state has no leader
I20250115 14:52:00.054087 1162 tablet_bootstrap.cc:654] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.054078 1164 ts_tablet_manager.cc:1428] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:00.054365 1164 tablet_bootstrap.cc:492] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.055167 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 0de6f6f014074c3d8775cb5e4121ccb0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:00.055379 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b60ff80679a470cb4bdced71cc14195 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.055409 1751 ts_tablet_manager.cc:1918] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.055531 1751 log.cc:1198] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/1f1df9dc743342b4b7bdb1a8d3fcd5b0
I20250115 14:52:00.055953 1751 ts_tablet_manager.cc:1939] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:00.056186 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7b4c900318f642499ecffcb268290262" } modify_external_catalogs: true
I20250115 14:52:00.056743 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7b4c900318f642499ecffcb268290262" } modify_external_catalogs: true
I20250115 14:52:00.057073 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 (table default.Table_fc422b5daf4a4770945aab3d34bc5d35 [id=c8b8a13b708841e2ad2017a4fa7d01d1]) successfully deleted
I20250115 14:52:00.053355 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet fe696560ba6b4a76ad013d9a3486e398 (table default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]). Expected version 1 got 0
I20250115 14:52:00.058248 1751 tablet_replica.cc:331] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:00.058393 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 5bc975b044dd439dbd1f8f9a82cce751 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bc975b044dd439dbd1f8f9a82cce751 already in progress: creating tablet
I20250115 14:52:00.058504 1751 raft_consensus.cc:2238] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.058662 1751 raft_consensus.cc:2267] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.058820 1750 raft_consensus.cc:491] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.058943 1750 raft_consensus.cc:513] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.059334 1751 ts_tablet_manager.cc:1905] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.059541 1750 leader_election.cc:290] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:00.060506 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de6f6f014074c3d8775cb5e4121ccb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.060593 1164 tablet_bootstrap.cc:654] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:00.061048 710 leader_election.cc:343] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d86075ae5b894d5986ac4bd538f4d42a
W20250115 14:52:00.062237 710 leader_election.cc:343] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: d86075ae5b894d5986ac4bd538f4d42a
I20250115 14:52:00.062374 710 leader_election.cc:304] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:00.062232 1166 ts_tablet_manager.cc:1918] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.062813 1750 raft_consensus.cc:2743] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:00.063022 1166 log.cc:1198] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/3e990a2920b542f69a8bdd022db6e7e2
I20250115 14:52:00.063778 1166 ts_tablet_manager.cc:1939] T 3e990a2920b542f69a8bdd022db6e7e2 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.064436 891 tablet_service.cc:1514] Processing DeleteTablet for tablet f847f92cd122423aa93f1afa701b0bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 83415f1c07ad495cabd287bdcd54dad4) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.064810 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 3e990a2920b542f69a8bdd022db6e7e2 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]) successfully deleted
W20250115 14:52:00.065734 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4294c49da1e044be993100b3cf5e2cea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4294c49da1e044be993100b3cf5e2cea already in progress: creating tablet
W20250115 14:52:00.066308 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6e616d78933e4f84b9ec2acc51450b18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e616d78933e4f84b9ec2acc51450b18 already in progress: creating tablet
I20250115 14:52:00.066608 1166 tablet_replica.cc:331] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.066833 1166 raft_consensus.cc:2238] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.067049 1166 raft_consensus.cc:2267] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.068467 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 629873c00550408798b00489ece4ea5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.070559 1751 ts_tablet_manager.cc:1918] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.070760 1751 log.cc:1198] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/75788f8eb36b446e927197f624fbfdf5
I20250115 14:52:00.071136 1751 ts_tablet_manager.cc:1939] T 75788f8eb36b446e927197f624fbfdf5 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:00.072101 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 75788f8eb36b446e927197f624fbfdf5 (table default.Table_875d7bd9588344008279ca89c1ad52be [id=7f3b54e6dae14efcb524f4f03c0a57b1]) successfully deleted
I20250115 14:52:00.072470 1751 tablet_replica.cc:331] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:00.072692 1751 raft_consensus.cc:2238] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.073482 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 9f4475441e4a467a8d593b742d2a5b80 (DEFAULT_TABLE table=default.Table_c763759bb01d449cbd6e2f851b303ec5 [id=4f4b4f8aa6364221b4a35c423a5631aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.073845 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4475441e4a467a8d593b742d2a5b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.073925 749 tablet_service.cc:1467] Processing CreateTablet for tablet 246720dbb4ed4216ac87fb4a3df79e2c (DEFAULT_TABLE table=default.Table_03c9410818db4ce190e6ca259f472d70 [id=8c0229e6f6884ea7be3417678bb27377]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.074301 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 246720dbb4ed4216ac87fb4a3df79e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.075050 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 246720dbb4ed4216ac87fb4a3df79e2c (DEFAULT_TABLE table=default.Table_03c9410818db4ce190e6ca259f472d70 [id=8c0229e6f6884ea7be3417678bb27377]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.075450 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 246720dbb4ed4216ac87fb4a3df79e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.077050 1162 tablet_bootstrap.cc:492] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:00.077198 1162 ts_tablet_manager.cc:1397] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20250115 14:52:00.078569 1162 raft_consensus.cc:357] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.079327 1162 raft_consensus.cc:383] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.079453 1162 raft_consensus.cc:738] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.079794 1162 consensus_queue.cc:260] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.080045 1732 raft_consensus.cc:491] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.080209 1732 raft_consensus.cc:513] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.080790 1732 leader_election.cc:290] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:00.081022 891 tablet_service.cc:1467] Processing CreateTablet for tablet 246720dbb4ed4216ac87fb4a3df79e2c (DEFAULT_TABLE table=default.Table_03c9410818db4ce190e6ca259f472d70 [id=8c0229e6f6884ea7be3417678bb27377]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.081379 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 246720dbb4ed4216ac87fb4a3df79e2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.081480 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9145bbaf3e234f079e355454f3ce28cb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:00.082131 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9145bbaf3e234f079e355454f3ce28cb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:00.082209 983 leader_election.cc:343] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:00.084066 984 leader_election.cc:343] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:00.084203 984 leader_election.cc:304] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:00.084177 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5da68f9f7cdb439daae3b29bc1e40438"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.084847 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5da68f9f7cdb439daae3b29bc1e40438 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.085819 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 8f8bae9867d745afb1dfdf47bcb881da
I20250115 14:52:00.086048 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c6d900c973594576916cf9d2b3a730b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:00.086431 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8bae9867d745afb1dfdf47bcb881da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.084364 1162 ts_tablet_manager.cc:1428] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20250115 14:52:00.086545 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6d900c973594576916cf9d2b3a730b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.087126 1728 raft_consensus.cc:2743] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:00.087594 1162 tablet_bootstrap.cc:492] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:00.087621 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f3c56dc604f2471083ae4303c707c7e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.088058 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3c56dc604f2471083ae4303c707c7e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.089148 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c0256da787104a42a1bc1e3946e11fd6" } modify_external_catalogs: true
I20250115 14:52:00.089287 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c0256da787104a42a1bc1e3946e11fd6" } modify_external_catalogs: true
I20250115 14:52:00.089634 1162 tablet_bootstrap.cc:654] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.091399 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 9145bbaf3e234f079e355454f3ce28cb (table default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]). Expected version 1 got 0
W20250115 14:52:00.091840 454 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 9145bbaf3e234f079e355454f3ce28cb (table default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9145bbaf3e234f079e355454f3ce28cb consensus state has no leader
I20250115 14:52:00.092437 1751 raft_consensus.cc:2267] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:00.092809 1698 raft_consensus.cc:668] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20250115 14:52:00.093482 1751 ts_tablet_manager.cc:1905] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.083745 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_de1856a934f84a1fa118e04a42ef373e" } modify_external_catalogs: true
I20250115 14:52:00.096961 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_de1856a934f84a1fa118e04a42ef373e" } modify_external_catalogs: true
I20250115 14:52:00.099475 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 9145bbaf3e234f079e355454f3ce28cb (table default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]). Expected version 1 got 0
I20250115 14:52:00.101011 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8bae9867d745afb1dfdf47bcb881da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.101303 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8e6652dab3bc46b7b4f1b66415398217" } modify_external_catalogs: true
I20250115 14:52:00.101461 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8e6652dab3bc46b7b4f1b66415398217" } modify_external_catalogs: true
I20250115 14:52:00.105195 1751 ts_tablet_manager.cc:1918] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.105754 1751 log.cc:1198] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6f07ef30cac2470d9b2f428338dd37a3
I20250115 14:52:00.106178 1751 ts_tablet_manager.cc:1939] T 6f07ef30cac2470d9b2f428338dd37a3 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:00.108042 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6f07ef30cac2470d9b2f428338dd37a3 (table default.Table_17b896568dc34b1aaf2b3e197d03fdc4 [id=b9d32cae6ebd42cea68dc15f454318ab]) successfully deleted
I20250115 14:52:00.107012 1162 tablet_bootstrap.cc:492] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:00.108537 1162 ts_tablet_manager.cc:1397] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.002s
I20250115 14:52:00.109617 1162 raft_consensus.cc:357] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.109869 1162 raft_consensus.cc:383] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.109983 1162 raft_consensus.cc:738] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.110272 1162 consensus_queue.cc:260] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.113566 1162 ts_tablet_manager.cc:1428] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:00.114044 1162 tablet_bootstrap.cc:492] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:00.115780 1162 tablet_bootstrap.cc:654] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.116004 1751 tablet_replica.cc:331] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:00.116564 448 master_service.cc:946] ReplaceTablet: received request to replace tablet d4c984613c7d45d5b7c513cd900c9558 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:00.116685 1751 raft_consensus.cc:2238] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.117401 1751 raft_consensus.cc:2267] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.120766 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4f1ce59e05054a009cd36e12db35d449
I20250115 14:52:00.121521 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cdbc0e16d4224a709c7c5aaf69918e64" } new_table_name: "default.Table_92acdab3d67543ae8f72d76b7013dbce" modify_external_catalogs: true
I20250115 14:52:00.125145 1168 tablet_bootstrap.cc:492] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:00.125756 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1ce59e05054a009cd36e12db35d449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:00.126271 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 01291d41a2b345b4bfd00e6be0fb0129 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01291d41a2b345b4bfd00e6be0fb0129 already in progress: creating tablet
I20250115 14:52:00.126825 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1ce59e05054a009cd36e12db35d449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.127506 749 tablet_service.cc:1467] Processing CreateTablet for tablet 398ac8af619c427287b3964443173864 (DEFAULT_TABLE table=default.Table_a26e212086b1478cb159097102934dd1 [id=4bb94d3cca0c4a14aad6957911139e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.127976 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 398ac8af619c427287b3964443173864. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.129957 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_53bd836a66f34fdea6b15e2c3122ad8e" } new_table_name: "default.Table_f97b1ee97b004066822fd87d21173ffa" modify_external_catalogs: true
I20250115 14:52:00.131688 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0e5d68124c7a4a0dbde272f4fb012cea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.132237 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e5d68124c7a4a0dbde272f4fb012cea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.133226 748 tablet_service.cc:1467] Processing CreateTablet for tablet 86c04fa23e7c40a5b5e07852e5d74a27 (DEFAULT_TABLE table=default.Table_702932472ce94b91b4535e0015a38be1 [id=04c521c944614e03be02fcb2ee8499c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.133387 741 tablet_service.cc:1467] Processing CreateTablet for tablet 4a79f880415a4b16aabdbb7cd989efdb (DEFAULT_TABLE table=default.Table_86cbb9218b1e442f8537cf2dd2d329af [id=f6e668ec62be4a1dbd638f0d323b114a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.133610 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86c04fa23e7c40a5b5e07852e5d74a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.133890 750 tablet_service.cc:1467] Processing CreateTablet for tablet a5e97279185541b992340c24ede01656 (DEFAULT_TABLE table=default.Table_55c2a1e317e34c3cabd92d742e6f2d6b [id=361ed20953ab4e79b1112ae76b95a423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.134260 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5e97279185541b992340c24ede01656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.136535 751 tablet_service.cc:1467] Processing CreateTablet for tablet e49dc97eea474252a5e0ef4af6e015a9 (DEFAULT_TABLE table=default.Table_fc4f396af4f54518b0232f10057dd159 [id=8502b5fe264f4fb5a89bd4a91949c90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.136824 1168 tablet_bootstrap.cc:654] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.136868 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e49dc97eea474252a5e0ef4af6e015a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.136909 743 tablet_service.cc:1467] Processing CreateTablet for tablet ded663ac1197454bbde3c4c2d526002d (DEFAULT_TABLE table=default.Table_e48cf46cf80d4d98a4955f201d993354 [id=a9793eda46fd4d07963603331438decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.137238 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded663ac1197454bbde3c4c2d526002d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.140431 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a79f880415a4b16aabdbb7cd989efdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.142000 1166 ts_tablet_manager.cc:1905] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.150729 890 tablet_service.cc:1467] Processing CreateTablet for tablet e49dc97eea474252a5e0ef4af6e015a9 (DEFAULT_TABLE table=default.Table_fc4f396af4f54518b0232f10057dd159 [id=8502b5fe264f4fb5a89bd4a91949c90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.150846 891 tablet_service.cc:1467] Processing CreateTablet for tablet 398ac8af619c427287b3964443173864 (DEFAULT_TABLE table=default.Table_a26e212086b1478cb159097102934dd1 [id=4bb94d3cca0c4a14aad6957911139e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.151257 884 tablet_service.cc:1467] Processing CreateTablet for tablet 86c04fa23e7c40a5b5e07852e5d74a27 (DEFAULT_TABLE table=default.Table_702932472ce94b91b4535e0015a38be1 [id=04c521c944614e03be02fcb2ee8499c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.151616 885 tablet_service.cc:1467] Processing CreateTablet for tablet ded663ac1197454bbde3c4c2d526002d (DEFAULT_TABLE table=default.Table_e48cf46cf80d4d98a4955f201d993354 [id=a9793eda46fd4d07963603331438decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.152037 876 tablet_service.cc:1467] Processing CreateTablet for tablet 4a79f880415a4b16aabdbb7cd989efdb (DEFAULT_TABLE table=default.Table_86cbb9218b1e442f8537cf2dd2d329af [id=f6e668ec62be4a1dbd638f0d323b114a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.154088 756 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8bae9867d745afb1dfdf47bcb881da (DEFAULT_TABLE table=default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.154456 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8bae9867d745afb1dfdf47bcb881da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.154408 882 tablet_service.cc:1467] Processing CreateTablet for tablet a5e97279185541b992340c24ede01656 (DEFAULT_TABLE table=default.Table_55c2a1e317e34c3cabd92d742e6f2d6b [id=361ed20953ab4e79b1112ae76b95a423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.156517 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8bae9867d745afb1dfdf47bcb881da (DEFAULT_TABLE table=default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.156904 877 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd94e4a7fef44f5be462e7a77e8a123 (DEFAULT_TABLE table=default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.157243 887 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4c2707915e4e97ad6c66cb249c6d9e (DEFAULT_TABLE table=default.Table_eb0e928b8265482990bb5fbf1072985f [id=dfcc6db7be294db4a0f620ab93714f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.161640 748 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4c2707915e4e97ad6c66cb249c6d9e (DEFAULT_TABLE table=default.Table_eb0e928b8265482990bb5fbf1072985f [id=dfcc6db7be294db4a0f620ab93714f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.162092 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4c2707915e4e97ad6c66cb249c6d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.163740 744 tablet_service.cc:1467] Processing CreateTablet for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 (DEFAULT_TABLE table=default.Table_233462856fc84da29a45b15193b81652 [id=c66ea162404c4eb0a012128385984399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.164067 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.165211 740 tablet_service.cc:1467] Processing CreateTablet for tablet 6308f68e41554470b7d1321c60d4eee9 (DEFAULT_TABLE table=default.Table_33f4705d7a1b474d82734fe4a80c0424 [id=b0ebc66f46364e51b73bdee697f00b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.165567 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6308f68e41554470b7d1321c60d4eee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.167642 752 tablet_service.cc:1467] Processing CreateTablet for tablet e244689cef6844e2935becd265eecad0 (DEFAULT_TABLE table=default.Table_53bd836a66f34fdea6b15e2c3122ad8e [id=1cdf416758254c679c3c785b101c65e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.169306 878 tablet_service.cc:1467] Processing CreateTablet for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 (DEFAULT_TABLE table=default.Table_233462856fc84da29a45b15193b81652 [id=c66ea162404c4eb0a012128385984399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.169723 892 tablet_service.cc:1467] Processing CreateTablet for tablet 6308f68e41554470b7d1321c60d4eee9 (DEFAULT_TABLE table=default.Table_33f4705d7a1b474d82734fe4a80c0424 [id=b0ebc66f46364e51b73bdee697f00b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.171648 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d5a5a92804cb462181e68e1cec02102b
I20250115 14:52:00.172242 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8c90e9dde49d43239bcb743145f4ef07" } new_table_name: "default.Table_49688b4238ca4abaac09f3cbdb738761" modify_external_catalogs: true
I20250115 14:52:00.172314 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_33a2bbb1ba7d426caf1eee40268cec01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.172832 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33a2bbb1ba7d426caf1eee40268cec01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.173544 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d4c984613c7d45d5b7c513cd900c9558
I20250115 14:52:00.173972 448 catalog_manager.cc:6596] ReplaceTablet: tablet d4c984613c7d45d5b7c513cd900c9558 deleted and replaced by tablet 1218f0bdb1e14bd78d76c6a06805f38c
I20250115 14:52:00.174242 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3afc89f290d048999c9654eeed8bb849" } new_table_name: "default.Table_2e2abefaccf54e469bacf8126edad0c4" modify_external_catalogs: true
I20250115 14:52:00.175452 743 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a5a92804cb462181e68e1cec02102b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.175634 739 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c984613c7d45d5b7c513cd900c9558 with delete_type TABLET_DATA_DELETED (replaced by tablet 1218f0bdb1e14bd78d76c6a06805f38c) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.175980 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_62b16563153946409ee7c59069520182"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:00.176421 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4239747374aa4d6d866961430786677c
I20250115 14:52:00.176813 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c984613c7d45d5b7c513cd900c9558 with delete_type TABLET_DATA_DELETED (replaced by tablet 1218f0bdb1e14bd78d76c6a06805f38c) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.177047 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d18e82c7d46c4394b6037413cd35cef6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:00.177635 874 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a5a92804cb462181e68e1cec02102b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.177780 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e244689cef6844e2935becd265eecad0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:00.176443 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62b16563153946409ee7c59069520182 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:00.179970 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d18e82c7d46c4394b6037413cd35cef6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.181367 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 4239747374aa4d6d866961430786677c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.183216 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_69b643bd59a847959f3f29e23f007498" } new_table_name: "default.Table_bb07dc16172246278fff8624127ab8bd" modify_external_catalogs: true
I20250115 14:52:00.184393 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3a0d97fa57484b9d8cb879fedda61612
I20250115 14:52:00.185127 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c9a1822ec7ae455ba4b2d5a1181d3eb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:00.185264 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0d97fa57484b9d8cb879fedda61612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:00.185580 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9a1822ec7ae455ba4b2d5a1181d3eb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.185662 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0d97fa57484b9d8cb879fedda61612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.187716 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4239747374aa4d6d866961430786677c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.188081 874 tablet_service.cc:1467] Processing CreateTablet for tablet e244689cef6844e2935becd265eecad0 (DEFAULT_TABLE table=default.Table_53bd836a66f34fdea6b15e2c3122ad8e [id=1cdf416758254c679c3c785b101c65e2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:00.190635 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e244689cef6844e2935becd265eecad0 consensus state has no leader
I20250115 14:52:00.191036 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9daeb4cb953b43a6bb9dc34b756d7716"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.191602 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9daeb4cb953b43a6bb9dc34b756d7716 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.193290 886 tablet_service.cc:1467] Processing CreateTablet for tablet 04daad461f0b433ba635df87439b5bc1 (DEFAULT_TABLE table=default.Table_c62ee465977947a19aba3314a3e2a3f7 [id=289c1cdb743b49b5893f551afab62d43]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:00.193413 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5265d5eeac83481ca97801e3475886da consensus state has no leader
I20250115 14:52:00.193795 889 tablet_service.cc:1467] Processing CreateTablet for tablet 36de948083a946eba1a96f276e4f4f91 (DEFAULT_TABLE table=default.Table_89373c553f9c407a9e205bfe0d1acf99 [id=26c53728ab22430b95ca322543118a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.194633 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_61bd6e7ab702444ba0919bfb8efb04f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.195062 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61bd6e7ab702444ba0919bfb8efb04f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:00.195974 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dff155a5c2b0479a90e74993de6f402d consensus state has no leader
I20250115 14:52:00.196161 752 tablet_service.cc:1467] Processing CreateTablet for tablet 04daad461f0b433ba635df87439b5bc1 (DEFAULT_TABLE table=default.Table_c62ee465977947a19aba3314a3e2a3f7 [id=289c1cdb743b49b5893f551afab62d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.196590 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04daad461f0b433ba635df87439b5bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.197899 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 4b90205ef15f470a99c013e7f2cf99c6 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:00.198096 750 tablet_service.cc:1467] Processing CreateTablet for tablet 36de948083a946eba1a96f276e4f4f91 (DEFAULT_TABLE table=default.Table_89373c553f9c407a9e205bfe0d1acf99 [id=26c53728ab22430b95ca322543118a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.199816 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36de948083a946eba1a96f276e4f4f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.200528 741 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd94e4a7fef44f5be462e7a77e8a123 (DEFAULT_TABLE table=default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.200862 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd94e4a7fef44f5be462e7a77e8a123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.203660 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e244689cef6844e2935becd265eecad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.208071 1166 ts_tablet_manager.cc:1918] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.208647 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e49dc97eea474252a5e0ef4af6e015a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.213537 1164 tablet_bootstrap.cc:492] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.213683 1164 ts_tablet_manager.cc:1397] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.159s user 0.020s sys 0.028s
I20250115 14:52:00.214465 1164 raft_consensus.cc:357] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.214677 1164 raft_consensus.cc:383] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.214776 1164 raft_consensus.cc:738] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.214989 1164 consensus_queue.cc:260] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.217327 1164 ts_tablet_manager.cc:1428] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:00.217614 1164 tablet_bootstrap.cc:492] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.219188 1164 tablet_bootstrap.cc:654] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.221117 888 tablet_service.cc:1467] Processing CreateTablet for tablet a300a76b8f7942dea648a20cbb3d5368 (DEFAULT_TABLE table=default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.221616 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 398ac8af619c427287b3964443173864. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.223007 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8bae9867d745afb1dfdf47bcb881da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.212941 741 tablet_service.cc:1467] Processing CreateTablet for tablet a300a76b8f7942dea648a20cbb3d5368 (DEFAULT_TABLE table=default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.222752 1166 log.cc:1198] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d86075ae5b894d5986ac4bd538f4d42a
I20250115 14:52:00.223664 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e50462626f914fb1b0f440c48f30e72d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.224025 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a300a76b8f7942dea648a20cbb3d5368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.228195 1166 ts_tablet_manager.cc:1939] T d86075ae5b894d5986ac4bd538f4d42a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.229020 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5e97279185541b992340c24ede01656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.229997 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4c2707915e4e97ad6c66cb249c6d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.230577 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36de948083a946eba1a96f276e4f4f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.233214 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a300a76b8f7942dea648a20cbb3d5368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.233361 1164 tablet_bootstrap.cc:492] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.233520 1164 ts_tablet_manager.cc:1397] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250115 14:52:00.233875 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86c04fa23e7c40a5b5e07852e5d74a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.234305 1164 raft_consensus.cc:357] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.234506 1164 raft_consensus.cc:383] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.234567 1164 raft_consensus.cc:738] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.234943 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd94e4a7fef44f5be462e7a77e8a123. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.234877 1164 consensus_queue.cc:260] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.236158 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.238198 1164 ts_tablet_manager.cc:1428] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:00.238565 1164 tablet_bootstrap.cc:492] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.238632 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6308f68e41554470b7d1321c60d4eee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.239641 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded663ac1197454bbde3c4c2d526002d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.240233 1164 tablet_bootstrap.cc:654] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.241397 1166 tablet_replica.cc:331] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.242004 1166 raft_consensus.cc:2238] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.242691 1166 raft_consensus.cc:2267] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.243990 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04daad461f0b433ba635df87439b5bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.245043 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a79f880415a4b16aabdbb7cd989efdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.245153 1166 ts_tablet_manager.cc:1905] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.258162 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d86075ae5b894d5986ac4bd538f4d42a (table default.Table_c5f3c652f95240ff9f080165af17f5e6 [id=54e9cab363084972abe02521b2cc19b8]) successfully deleted
W20250115 14:52:00.258496 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 43605c995df4400e91139199c9514bfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43605c995df4400e91139199c9514bfd already in progress: creating tablet
I20250115 14:52:00.262885 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b25dce523e724066b203e6111cad8111 with delete_type TABLET_DATA_DELETED (replaced by tablet 0de6f6f014074c3d8775cb5e4121ccb0) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:00.265453 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a48227ec5aa846c3ac63bd29ed38be25 consensus state has no leader
I20250115 14:52:00.268170 1750 raft_consensus.cc:491] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:52:00.268283 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 377d711f052545ddbe499afe7ba3de28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 377d711f052545ddbe499afe7ba3de28 already in progress: creating tablet
I20250115 14:52:00.268337 1750 raft_consensus.cc:513] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.268965 1750 leader_election.cc:290] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:00.269491 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet d97c2f443ab040f9bc1924f45379165e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.271328 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4b90205ef15f470a99c013e7f2cf99c6
I20250115 14:52:00.271790 446 catalog_manager.cc:6596] ReplaceTablet: tablet 4b90205ef15f470a99c013e7f2cf99c6 deleted and replaced by tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd
I20250115 14:52:00.273260 1732 raft_consensus.cc:491] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.273407 1732 raft_consensus.cc:513] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.274003 1732 leader_election.cc:290] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:00.274515 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7233ebf3bae1462d8eeb3669b39944c0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:00.275023 1751 ts_tablet_manager.cc:1905] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.279493 1166 ts_tablet_manager.cc:1918] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:00.279479 984 leader_election.cc:343] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:00.280587 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8bae9867d745afb1dfdf47bcb881da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.280614 1166 log.cc:1198] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9aeca9975f1945ad956f47db1ea16a73
I20250115 14:52:00.280982 1166 ts_tablet_manager.cc:1939] T 9aeca9975f1945ad956f47db1ea16a73 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.281343 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 398ac8af619c427287b3964443173864 (DEFAULT_TABLE table=default.Table_a26e212086b1478cb159097102934dd1 [id=4bb94d3cca0c4a14aad6957911139e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.283613 1698 raft_consensus.cc:491] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.284129 1750 raft_consensus.cc:491] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.284317 1164 tablet_bootstrap.cc:492] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.284526 1164 ts_tablet_manager.cc:1397] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.010s
I20250115 14:52:00.284507 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8bae9867d745afb1dfdf47bcb881da (DEFAULT_TABLE table=default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.284369 1750 raft_consensus.cc:513] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.284799 1166 tablet_replica.cc:331] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.284902 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9aeca9975f1945ad956f47db1ea16a73 (table default.Table_cf9f4b392dce4957a5e5d4ba0cb9f666 [id=914642e9e7974554a76a94e205db108f]) successfully deleted
I20250115 14:52:00.284979 1166 raft_consensus.cc:2238] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.284992 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd94e4a7fef44f5be462e7a77e8a123 (DEFAULT_TABLE table=default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.285082 1750 leader_election.cc:290] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:00.285151 1166 raft_consensus.cc:2267] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.285494 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 (DEFAULT_TABLE table=default.Table_233462856fc84da29a45b15193b81652 [id=c66ea162404c4eb0a012128385984399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.285873 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c984613c7d45d5b7c513cd900c9558 with delete_type TABLET_DATA_DELETED (replaced by tablet 1218f0bdb1e14bd78d76c6a06805f38c) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:00.286435 710 leader_election.cc:343] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e50462626f914fb1b0f440c48f30e72d
I20250115 14:52:00.286898 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 4239747374aa4d6d866961430786677c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.287220 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 04daad461f0b433ba635df87439b5bc1 (DEFAULT_TABLE table=default.Table_c62ee465977947a19aba3314a3e2a3f7 [id=289c1cdb743b49b5893f551afab62d43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.287259 1166 ts_tablet_manager.cc:1905] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.287959 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 36de948083a946eba1a96f276e4f4f91 (DEFAULT_TABLE table=default.Table_89373c553f9c407a9e205bfe0d1acf99 [id=26c53728ab22430b95ca322543118a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.287253 1164 raft_consensus.cc:357] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.288166 1164 raft_consensus.cc:383] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.288259 1164 raft_consensus.cc:738] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.288628 1164 consensus_queue.cc:260] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.288786 1025 tablet_service.cc:1467] Processing CreateTablet for tablet a5e97279185541b992340c24ede01656 (DEFAULT_TABLE table=default.Table_55c2a1e317e34c3cabd92d742e6f2d6b [id=361ed20953ab4e79b1112ae76b95a423]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.289841 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 4a79f880415a4b16aabdbb7cd989efdb (DEFAULT_TABLE table=default.Table_86cbb9218b1e442f8537cf2dd2d329af [id=f6e668ec62be4a1dbd638f0d323b114a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.290191 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1ce59e05054a009cd36e12db35d449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.290609 1164 ts_tablet_manager.cc:1428] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:52:00.290946 1164 tablet_bootstrap.cc:492] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.291604 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1839edd0d49b467d8bb31b3001ff89eb" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:00.292722 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 398ac8af619c427287b3964443173864. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.292877 1164 tablet_bootstrap.cc:654] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.295212 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ca710cea154be5b0f57ace4af4168b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.296556 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 86c04fa23e7c40a5b5e07852e5d74a27 (DEFAULT_TABLE table=default.Table_702932472ce94b91b4535e0015a38be1 [id=04c521c944614e03be02fcb2ee8499c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.297747 1016 tablet_service.cc:1467] Processing CreateTablet for tablet e49dc97eea474252a5e0ef4af6e015a9 (DEFAULT_TABLE table=default.Table_fc4f396af4f54518b0232f10057dd159 [id=8502b5fe264f4fb5a89bd4a91949c90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.299281 1029 tablet_service.cc:1467] Processing CreateTablet for tablet ded663ac1197454bbde3c4c2d526002d (DEFAULT_TABLE table=default.Table_e48cf46cf80d4d98a4955f201d993354 [id=a9793eda46fd4d07963603331438decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.299487 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d1198c17efa04e4ea060b6d89580342a" } modify_external_catalogs: true
I20250115 14:52:00.299660 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d1198c17efa04e4ea060b6d89580342a" } modify_external_catalogs: true
I20250115 14:52:00.301036 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0d97fa57484b9d8cb879fedda61612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.301119 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 6308f68e41554470b7d1321c60d4eee9 (DEFAULT_TABLE table=default.Table_33f4705d7a1b474d82734fe4a80c0424 [id=b0ebc66f46364e51b73bdee697f00b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.301223 1014 tablet_service.cc:1467] Processing CreateTablet for tablet e244689cef6844e2935becd265eecad0 (DEFAULT_TABLE table=default.Table_53bd836a66f34fdea6b15e2c3122ad8e [id=1cdf416758254c679c3c785b101c65e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.301662 1698 raft_consensus.cc:513] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.302322 1698 leader_election.cc:290] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:00.301472 1013 tablet_service.cc:1467] Processing CreateTablet for tablet a300a76b8f7942dea648a20cbb3d5368 (DEFAULT_TABLE table=default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.302809 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 2e4c2707915e4e97ad6c66cb249c6d9e (DEFAULT_TABLE table=default.Table_eb0e928b8265482990bb5fbf1072985f [id=dfcc6db7be294db4a0f620ab93714f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.303143 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet d5a5a92804cb462181e68e1cec02102b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.303444 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b90205ef15f470a99c013e7f2cf99c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.303678 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.303925 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04daad461f0b433ba635df87439b5bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.304119 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a300a76b8f7942dea648a20cbb3d5368. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.304401 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e244689cef6844e2935becd265eecad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.304597 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e4c2707915e4e97ad6c66cb249c6d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.304718 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629873c00550408798b00489ece4ea5f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:00.305320 984 leader_election.cc:343] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:00.305680 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ded663ac1197454bbde3c4c2d526002d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.306109 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6308f68e41554470b7d1321c60d4eee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.306514 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e49dc97eea474252a5e0ef4af6e015a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.307178 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86c04fa23e7c40a5b5e07852e5d74a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.310761 1698 raft_consensus.cc:491] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.310916 1698 raft_consensus.cc:513] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.311964 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c984613c7d45d5b7c513cd900c9558" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:00.312446 984 leader_election.cc:343] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:00.318346 1166 ts_tablet_manager.cc:1918] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.318483 1164 tablet_bootstrap.cc:492] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.318636 1166 log.cc:1198] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/6b60ff80679a470cb4bdced71cc14195
I20250115 14:52:00.318701 1164 ts_tablet_manager.cc:1397] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.002s
I20250115 14:52:00.319305 1166 ts_tablet_manager.cc:1939] T 6b60ff80679a470cb4bdced71cc14195 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.319571 1164 raft_consensus.cc:357] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.319830 1164 raft_consensus.cc:383] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.319947 1164 raft_consensus.cc:738] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.320267 1164 consensus_queue.cc:260] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.322808 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 6b60ff80679a470cb4bdced71cc14195 (table default.Table_c088519ba27c457983f341363f8a109e [id=201c214e2eab4873af2095e83dc88b23]) successfully deleted
I20250115 14:52:00.312201 1698 leader_election.cc:290] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:00.323387 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd94e4a7fef44f5be462e7a77e8a123. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:00.325193 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 8f8bae9867d745afb1dfdf47bcb881da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f8bae9867d745afb1dfdf47bcb881da already in progress: creating tablet
I20250115 14:52:00.325294 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b9ce445d334842c8a6bacad9f693feda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.325616 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0e5d68124c7a4a0dbde272f4fb012cea" } modify_external_catalogs: true
I20250115 14:52:00.325796 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0e5d68124c7a4a0dbde272f4fb012cea" } modify_external_catalogs: true
I20250115 14:52:00.327502 1166 tablet_replica.cc:331] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.328159 1166 raft_consensus.cc:2238] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.328485 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8bae9867d745afb1dfdf47bcb881da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.328742 1166 raft_consensus.cc:2267] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.329653 1164 ts_tablet_manager.cc:1428] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
W20250115 14:52:00.329931 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4f1ce59e05054a009cd36e12db35d449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1ce59e05054a009cd36e12db35d449 already in progress: creating tablet
I20250115 14:52:00.337042 741 tablet_service.cc:1467] Processing CreateTablet for tablet a48a0af9422f43cf86531665c0fa1aee (DEFAULT_TABLE table=default.Table_c6d900c973594576916cf9d2b3a730b0 [id=e6815e32e72f46dd95661c2e68069d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.337399 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48a0af9422f43cf86531665c0fa1aee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.339908 1167 tablet_replica.cc:331] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:00.340126 1167 raft_consensus.cc:2238] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.340283 1167 raft_consensus.cc:2267] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.340864 1167 ts_tablet_manager.cc:1905] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.342042 1782 raft_consensus.cc:491] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.344144 752 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2d55a4e32448fbb98f704dbf1657b9 (DEFAULT_TABLE table=default.Table_f3c56dc604f2471083ae4303c707c7e3 [id=03cd191267784fac938c73c9b363855a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.344513 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2d55a4e32448fbb98f704dbf1657b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.346057 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b90205ef15f470a99c013e7f2cf99c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.347165 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7233ebf3bae1462d8eeb3669b39944c0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:00.348580 748 tablet_service.cc:1467] Processing CreateTablet for tablet 18f23815196047628d562da7312b010f (DEFAULT_TABLE table=default.Table_af26e85d3ff84c6296a22baee6440af2 [id=bd973eb84bcb426fa3e5963b7cb949dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.348706 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629873c00550408798b00489ece4ea5f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:00.348721 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c984613c7d45d5b7c513cd900c9558" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:00.348934 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f23815196047628d562da7312b010f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.348923 913 raft_consensus.cc:2463] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:00.349169 1782 raft_consensus.cc:513] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.352541 756 tablet_service.cc:1467] Processing CreateTablet for tablet 48b15de9bf8943cdb9e5abd943484993 (DEFAULT_TABLE table=default.Table_a7e733ef2d844d238685b7d25b53c151 [id=c40fb10d4df34d58a790b3d4e39b48e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.352571 881 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2d55a4e32448fbb98f704dbf1657b9 (DEFAULT_TABLE table=default.Table_f3c56dc604f2471083ae4303c707c7e3 [id=03cd191267784fac938c73c9b363855a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.352859 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b15de9bf8943cdb9e5abd943484993. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.354450 880 tablet_service.cc:1467] Processing CreateTablet for tablet 18f23815196047628d562da7312b010f (DEFAULT_TABLE table=default.Table_af26e85d3ff84c6296a22baee6440af2 [id=bd973eb84bcb426fa3e5963b7cb949dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.355851 875 tablet_service.cc:1467] Processing CreateTablet for tablet 48b15de9bf8943cdb9e5abd943484993 (DEFAULT_TABLE table=default.Table_a7e733ef2d844d238685b7d25b53c151 [id=c40fb10d4df34d58a790b3d4e39b48e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.359251 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4f89ec0a5d68408c897f43ab39df62b1 (DEFAULT_TABLE table=default.Table_d1198c17efa04e4ea060b6d89580342a [id=5e381e3344a34b53a930fcdfc8c8fa25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.359591 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f89ec0a5d68408c897f43ab39df62b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.365912 740 tablet_service.cc:1467] Processing CreateTablet for tablet abefc717e0eb41b18725ef0c73154fdd (DEFAULT_TABLE table=default.Table_014f98e470c94ba4b3446630153a9e75 [id=3be2f53f8e374ee6aa37100f87b50b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.366251 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abefc717e0eb41b18725ef0c73154fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.366575 754 tablet_service.cc:1467] Processing CreateTablet for tablet 2560d42c15c64b4c9c0cfe95706b4d9e (DEFAULT_TABLE table=default.Table_99a5f2ca224f459dbf6bd4d4ec255457 [id=92ab45b4bf154e789e3a9b33c77d38c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.366719 744 tablet_service.cc:1467] Processing CreateTablet for tablet ed777d7496514ec5986a7e6f53d532b1 (DEFAULT_TABLE table=default.Table_56da29ad1fe24546bc5c6a4a17a774e7 [id=1b1cf9caa0194880af26c65d6f7bcbf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.366925 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2560d42c15c64b4c9c0cfe95706b4d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.367260 743 tablet_service.cc:1467] Processing CreateTablet for tablet e6978223c1984a38bec5f2d06a9cf457 (DEFAULT_TABLE table=default.Table_f9a5e1c80cc147cf861197fac6765b01 [id=e03ea31f74914ceba6e5a36ba1b30dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.367413 746 tablet_service.cc:1467] Processing CreateTablet for tablet d1259256b7744e739581125611abbe67 (DEFAULT_TABLE table=default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.367646 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6978223c1984a38bec5f2d06a9cf457. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.368412 745 tablet_service.cc:1467] Processing CreateTablet for tablet 1218f0bdb1e14bd78d76c6a06805f38c (DEFAULT_TABLE table=default.Table_272f9650cb3142f781f1d4d7c4769dfe [id=2fb684a787e4420aa9218b369eba5cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.368569 755 tablet_service.cc:1467] Processing CreateTablet for tablet 3f541e916d264738842cd965c7a2a5ec (DEFAULT_TABLE table=default.Table_33a2bbb1ba7d426caf1eee40268cec01 [id=8d19600fb0354c028cafa03a1c87e5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.368748 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1218f0bdb1e14bd78d76c6a06805f38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.368754 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4adf1353bd554d8294a7f570931095d2
I20250115 14:52:00.369057 739 tablet_service.cc:1467] Processing CreateTablet for tablet 72a97d102c174bc3a5e3a3c5762033b3 (DEFAULT_TABLE table=default.Table_0a8e879b253d48d3b02055cb2bd9663e [id=e07c949059424311bb6faf43de5dd443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.369386 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f541e916d264738842cd965c7a2a5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.369756 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0cae4c9228c640e495507e8f7ad9f544 (DEFAULT_TABLE table=default.Table_dae457edf44a40a69a40030ae24a26ff [id=0dd965b7347c439d9abcc56446730332]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.370136 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cae4c9228c640e495507e8f7ad9f544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.372488 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed777d7496514ec5986a7e6f53d532b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.372679 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4f89ec0a5d68408c897f43ab39df62b1
I20250115 14:52:00.375380 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1259256b7744e739581125611abbe67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.364811 1838 consensus_queue.cc:1035] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20250115 14:52:00.380584 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a97d102c174bc3a5e3a3c5762033b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.382453 893 tablet_service.cc:1467] Processing CreateTablet for tablet abefc717e0eb41b18725ef0c73154fdd (DEFAULT_TABLE table=default.Table_014f98e470c94ba4b3446630153a9e75 [id=3be2f53f8e374ee6aa37100f87b50b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.387212 874 tablet_service.cc:1467] Processing CreateTablet for tablet ed777d7496514ec5986a7e6f53d532b1 (DEFAULT_TABLE table=default.Table_56da29ad1fe24546bc5c6a4a17a774e7 [id=1b1cf9caa0194880af26c65d6f7bcbf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.387579 879 tablet_service.cc:1467] Processing CreateTablet for tablet 4f89ec0a5d68408c897f43ab39df62b1 (DEFAULT_TABLE table=default.Table_d1198c17efa04e4ea060b6d89580342a [id=5e381e3344a34b53a930fcdfc8c8fa25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.393755 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89ec0a5d68408c897f43ab39df62b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.395609 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5e97279185541b992340c24ede01656. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:00.400807 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9e95d09efd2e44cc81cdcbf87ee453fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e95d09efd2e44cc81cdcbf87ee453fe already in progress: creating tablet
W20250115 14:52:00.401095 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 742792be08444792892bd9cd7336dda3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 742792be08444792892bd9cd7336dda3 already in progress: creating tablet
I20250115 14:52:00.404775 889 tablet_service.cc:1467] Processing CreateTablet for tablet 72a97d102c174bc3a5e3a3c5762033b3 (DEFAULT_TABLE table=default.Table_0a8e879b253d48d3b02055cb2bd9663e [id=e07c949059424311bb6faf43de5dd443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.406514 1164 tablet_bootstrap.cc:492] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.408181 1164 tablet_bootstrap.cc:654] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.408388 883 tablet_service.cc:1467] Processing CreateTablet for tablet a48a0af9422f43cf86531665c0fa1aee (DEFAULT_TABLE table=default.Table_c6d900c973594576916cf9d2b3a730b0 [id=e6815e32e72f46dd95661c2e68069d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.411338 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_707a274426cd4d37a8ee193d73a0e171"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.411909 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_707a274426cd4d37a8ee193d73a0e171 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.413232 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c763759bb01d449cbd6e2f851b303ec5" } modify_external_catalogs: true
I20250115 14:52:00.413755 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9118acb2bd5e43e9b5d4b0d2379fadfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.414304 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9118acb2bd5e43e9b5d4b0d2379fadfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.414422 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4deac3b7f7124b6aa49514a17811b1c3" } modify_external_catalogs: true
I20250115 14:52:00.414500 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f7332a834b6a46018bacf978fc59830b" } modify_external_catalogs: true
I20250115 14:52:00.414772 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a2eb9b62487c4908880b44b9915e4cf7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:00.415100 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_02294519a7244bd6a9f4741db4a92d58" } new_table_name: "default.Table_1da7201c533742a28cf8ad8777ecfb28" modify_external_catalogs: true
W20250115 14:52:00.415262 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2eb9b62487c4908880b44b9915e4cf7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.415896 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c230050c7eab42e7a66a8132c7421def" } new_table_name: "default.Table_dd1b72b0118645adb63b44442697414f" modify_external_catalogs: true
I20250115 14:52:00.416343 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c6bb6ec57a9240c48a9f5607865e62c2" } new_table_name: "default.Table_c2aefc9f5c604090bb4bd4f2326991ba" modify_external_catalogs: true
I20250115 14:52:00.416343 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_38b74aa452d14ca5b22b7259f85af561"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.417013 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38b74aa452d14ca5b22b7259f85af561 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.420516 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4deac3b7f7124b6aa49514a17811b1c3" } modify_external_catalogs: true
I20250115 14:52:00.421396 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f7332a834b6a46018bacf978fc59830b" } modify_external_catalogs: true
I20250115 14:52:00.425200 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c763759bb01d449cbd6e2f851b303ec5" } modify_external_catalogs: true
I20250115 14:52:00.426013 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de6f6f014074c3d8775cb5e4121ccb0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:00.426203 746 tablet_service.cc:1467] Processing CreateTablet for tablet 4adf1353bd554d8294a7f570931095d2 (DEFAULT_TABLE table=default.Table_0e5d68124c7a4a0dbde272f4fb012cea [id=1c9959ef09ed4988baf48112f0280303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.426649 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4adf1353bd554d8294a7f570931095d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.429973 752 tablet_service.cc:1467] Processing CreateTablet for tablet e910c387e6a24caf95e14b57c8b83e6f (DEFAULT_TABLE table=default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.430513 741 tablet_service.cc:1467] Processing CreateTablet for tablet d6f2bc62621d402cacfcbe2eb74173f9 (DEFAULT_TABLE table=default.Table_62b16563153946409ee7c59069520182 [id=3249c65f52624113be96f7ef7b828854]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.431054 745 tablet_service.cc:1467] Processing CreateTablet for tablet ba33eb3f5a40450db5cf115fbe8a17e7 (DEFAULT_TABLE table=default.Table_a584af21c70545b8a2d3690c20422f1f [id=f8c4ca50220a4dbf9b9480213d836e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.408416 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 4adf1353bd554d8294a7f570931095d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.431720 739 tablet_service.cc:1467] Processing CreateTablet for tablet cc9ce226d3ba44d6908257bbd0df3a12 (DEFAULT_TABLE table=default.Table_2c94d0bb9dd54c5195b1fcbf7cec15fd [id=ceccf8508c5047c2b7a63788dd81a66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.432157 1168 tablet_bootstrap.cc:492] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:00.432394 1168 ts_tablet_manager.cc:1397] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.307s user 0.014s sys 0.002s
I20250115 14:52:00.433212 1168 raft_consensus.cc:357] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.433413 1168 raft_consensus.cc:383] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.433351 876 tablet_service.cc:1467] Processing CreateTablet for tablet 0cae4c9228c640e495507e8f7ad9f544 (DEFAULT_TABLE table=default.Table_dae457edf44a40a69a40030ae24a26ff [id=0dd965b7347c439d9abcc56446730332]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.433494 1168 raft_consensus.cc:738] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.433720 1168 consensus_queue.cc:260] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.434324 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2d55a4e32448fbb98f704dbf1657b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.434397 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e910c387e6a24caf95e14b57c8b83e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.434895 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba33eb3f5a40450db5cf115fbe8a17e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.435429 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6f2bc62621d402cacfcbe2eb74173f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.435703 1782 leader_election.cc:290] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:00.435853 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9ce226d3ba44d6908257bbd0df3a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.436149 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b15de9bf8943cdb9e5abd943484993. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.440064 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48a0af9422f43cf86531665c0fa1aee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.450407 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f89ec0a5d68408c897f43ab39df62b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.452455 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f23815196047628d562da7312b010f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:00.458173 848 leader_election.cc:343] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:00.468709 887 tablet_service.cc:1467] Processing CreateTablet for tablet d1259256b7744e739581125611abbe67 (DEFAULT_TABLE table=default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.469254 884 tablet_service.cc:1467] Processing CreateTablet for tablet 2560d42c15c64b4c9c0cfe95706b4d9e (DEFAULT_TABLE table=default.Table_99a5f2ca224f459dbf6bd4d4ec255457 [id=92ab45b4bf154e789e3a9b33c77d38c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.474624 1698 consensus_queue.cc:1035] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.121s
I20250115 14:52:00.474880 1164 tablet_bootstrap.cc:492] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.475015 1164 ts_tablet_manager.cc:1397] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.002s
I20250115 14:52:00.475235 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abefc717e0eb41b18725ef0c73154fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.475780 1164 raft_consensus.cc:357] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.475968 1164 raft_consensus.cc:383] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.476068 1164 raft_consensus.cc:738] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.476284 1164 consensus_queue.cc:260] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.479061 1018 tablet_service.cc:1467] Processing CreateTablet for tablet a48a0af9422f43cf86531665c0fa1aee (DEFAULT_TABLE table=default.Table_c6d900c973594576916cf9d2b3a730b0 [id=e6815e32e72f46dd95661c2e68069d5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.479809 1164 ts_tablet_manager.cc:1428] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20250115 14:52:00.474552 983 leader_election.cc:343] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:00.482331 1164 tablet_bootstrap.cc:492] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:00.485044 848 consensus_peers.cc:487] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:00.485805 1164 tablet_bootstrap.cc:654] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.486578 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2560d42c15c64b4c9c0cfe95706b4d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.485044 1168 ts_tablet_manager.cc:1428] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.042s user 0.006s sys 0.003s
I20250115 14:52:00.487144 983 leader_election.cc:304] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:00.487498 983 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 136421us
W20250115 14:52:00.488014 983 leader_election.cc:343] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:00.488135 983 leader_election.cc:304] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:00.488546 983 leader_election.cc:304] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:00.490011 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36de948083a946eba1a96f276e4f4f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.490109 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b90205ef15f470a99c013e7f2cf99c6 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.491429 1166 ts_tablet_manager.cc:1905] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.497292 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e034d7761a5247c79414b7112a1c97b2
I20250115 14:52:00.498240 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6b196b51d11442e5be9d5a3a2f6f3484"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.498731 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5937d6ad0494734a402fdc594eb9415 consensus state has no leader
W20250115 14:52:00.498831 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b196b51d11442e5be9d5a3a2f6f3484 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:00.499099 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6f9262e808b4589a21257ed3fb0363b consensus state has no leader
I20250115 14:52:00.499205 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_68042c9611eb4780a797c5bb2aec65e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:00.499497 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9145bbaf3e234f079e355454f3ce28cb
W20250115 14:52:00.499707 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68042c9611eb4780a797c5bb2aec65e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.501109 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9f4475441e4a467a8d593b742d2a5b80
I20250115 14:52:00.498831 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_99a5f2ca224f459dbf6bd4d4ec255457" } new_table_name: "default.Table_892cf0050cb245a7af3e84a893239fba" modify_external_catalogs: true
I20250115 14:52:00.502285 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dd6e49ce9dc4402b88b42b9ed72e08eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.502735 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd6e49ce9dc4402b88b42b9ed72e08eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.503765 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f97b1ee97b004066822fd87d21173ffa" } new_table_name: "default.Table_dfd7391e00dc4815aedb1a4c64eba86f" modify_external_catalogs: true
I20250115 14:52:00.504415 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1ee3b83d4fee4651b3979b44d4903db6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.504855 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ee3b83d4fee4651b3979b44d4903db6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.505272 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f8719599d09f457681ab1c5bbf617baa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.506525 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8719599d09f457681ab1c5bbf617baa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:00.508971 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 338c0f6f39fd4335bbcda733b28b8bf8 consensus state has no leader
I20250115 14:52:00.511165 1732 raft_consensus.cc:2798] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:00.512073 1698 raft_consensus.cc:2743] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:00.514083 750 tablet_service.cc:1514] Processing DeleteTablet for tablet e034d7761a5247c79414b7112a1c97b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.515504 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 18f23815196047628d562da7312b010f (DEFAULT_TABLE table=default.Table_af26e85d3ff84c6296a22baee6440af2 [id=bd973eb84bcb426fa3e5963b7cb949dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.517115 1728 raft_consensus.cc:2743] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:00.516844 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2d55a4e32448fbb98f704dbf1657b9 (DEFAULT_TABLE table=default.Table_f3c56dc604f2471083ae4303c707c7e3 [id=03cd191267784fac938c73c9b363855a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.519502 1166 ts_tablet_manager.cc:1918] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:00.519853 1166 log.cc:1198] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d5a5a92804cb462181e68e1cec02102b
I20250115 14:52:00.520196 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ae2b381e4eb84e6788f5d29ffc7a72f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.520787 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae2b381e4eb84e6788f5d29ffc7a72f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.524650 1166 ts_tablet_manager.cc:1939] T d5a5a92804cb462181e68e1cec02102b P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.528982 1166 tablet_replica.cc:331] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.529556 1166 raft_consensus.cc:2238] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.530400 1166 raft_consensus.cc:2267] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:00.532657 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e244689cef6844e2935becd265eecad0 consensus state has no leader
I20250115 14:52:00.534080 1166 ts_tablet_manager.cc:1905] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.535176 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 48b15de9bf8943cdb9e5abd943484993 (DEFAULT_TABLE table=default.Table_a7e733ef2d844d238685b7d25b53c151 [id=c40fb10d4df34d58a790b3d4e39b48e5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:00.535388 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2560d42c15c64b4c9c0cfe95706b4d9e consensus state has no leader
I20250115 14:52:00.535622 1732 raft_consensus.cc:491] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:00.535738 1732 raft_consensus.cc:3054] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:00.544323 1808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 14:52:00.544744 1808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 14:52:00.544894 1808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250115 14:52:00.546825 886 tablet_service.cc:1467] Processing CreateTablet for tablet e6978223c1984a38bec5f2d06a9cf457 (DEFAULT_TABLE table=default.Table_f9a5e1c80cc147cf861197fac6765b01 [id=e03ea31f74914ceba6e5a36ba1b30dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.551375 1164 tablet_bootstrap.cc:492] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.553460 1164 ts_tablet_manager.cc:1397] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.000s
W20250115 14:52:00.557791 1808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250115 14:52:00.557984 1808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 14:52:00.558099 1808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250115 14:52:00.558213 1808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250115 14:52:00.561825 1750 raft_consensus.cc:491] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.561959 1750 raft_consensus.cc:513] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.562597 1750 leader_election.cc:290] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:00.561116 1164 raft_consensus.cc:357] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.556747 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3f541e916d264738842cd965c7a2a5ec (DEFAULT_TABLE table=default.Table_33a2bbb1ba7d426caf1eee40268cec01 [id=8d19600fb0354c028cafa03a1c87e5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.565819 1164 raft_consensus.cc:383] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.566969 1164 raft_consensus.cc:738] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.568917 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a48a0af9422f43cf86531665c0fa1aee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.572477 1166 ts_tablet_manager.cc:1918] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:00.571079 1164 consensus_queue.cc:260] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.573755 1808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:45653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=40195
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4191
ASAN enabled
I20250115 14:52:00.574371 1166 log.cc:1198] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/4239747374aa4d6d866961430786677c
I20250115 14:52:00.576283 1808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20250115 14:52:00.578208 848 consensus_peers.cc:487] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:00.578980 1808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 14:52:00.579777 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d5a5a92804cb462181e68e1cec02102b (table default.Table_c0256da787104a42a1bc1e3946e11fd6 [id=22c418c3ac6648c687a8ae14d439aea5]) successfully deleted
W20250115 14:52:00.580049 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d4c984613c7d45d5b7c513cd900c9558 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4c984613c7d45d5b7c513cd900c9558 already in progress: creating tablet
I20250115 14:52:00.580421 1750 raft_consensus.cc:491] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.580566 1750 raft_consensus.cc:513] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.581162 1750 leader_election.cc:290] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:00.581444 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9145bbaf3e234f079e355454f3ce28cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.582614 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4475441e4a467a8d593b742d2a5b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.582828 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9e2c053d624b6ba71b031a591475a4 with delete_type TABLET_DATA_DELETED (replaced by tablet d5ffa997ca6c46559830b1dedf2d1a66) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.583137 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1259256b7744e739581125611abbe67. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:00.583348 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3a0d97fa57484b9d8cb879fedda61612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a0d97fa57484b9d8cb879fedda61612 already in progress: creating tablet
I20250115 14:52:00.584820 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 4f89ec0a5d68408c897f43ab39df62b1 (DEFAULT_TABLE table=default.Table_d1198c17efa04e4ea060b6d89580342a [id=5e381e3344a34b53a930fcdfc8c8fa25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.588820 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b15de9bf8943cdb9e5abd943484993. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.588423 1164 ts_tablet_manager.cc:1428] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20250115 14:52:00.588248 1166 ts_tablet_manager.cc:1939] T 4239747374aa4d6d866961430786677c P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.593964 1164 tablet_bootstrap.cc:492] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.596866 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d1345e860a054b708e11f2f64347a3d7" } modify_external_catalogs: true
I20250115 14:52:00.597124 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d1345e860a054b708e11f2f64347a3d7" } modify_external_catalogs: true
I20250115 14:52:00.598989 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18f23815196047628d562da7312b010f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:00.602128 710 leader_election.cc:343] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 50ca710cea154be5b0f57ace4af4168b
I20250115 14:52:00.602710 739 tablet_service.cc:1467] Processing CreateTablet for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c (DEFAULT_TABLE table=default.Table_c9a1822ec7ae455ba4b2d5a1181d3eb1 [id=b085d42488df4fee9c6da3f98b0a163a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.603080 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.605239 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_2e2abefaccf54e469bacf8126edad0c4 [id=855145809143431d81790dd36147a77b]). Expected version 2 got 0
I20250115 14:52:00.606969 750 tablet_service.cc:1467] Processing CreateTablet for tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd (DEFAULT_TABLE table=default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.607410 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.607563 743 tablet_service.cc:1467] Processing CreateTablet for tablet 6203efb862ea4962a6d09f3e4306a92f (DEFAULT_TABLE table=default.Table_d18e82c7d46c4394b6037413cd35cef6 [id=530bcb9252fa421b8df0f5932117a36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.607939 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6203efb862ea4962a6d09f3e4306a92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.608563 752 tablet_service.cc:1467] Processing CreateTablet for tablet d03eb225fad7459a955a3ec5e0002be1 (DEFAULT_TABLE table=default.Table_9daeb4cb953b43a6bb9dc34b756d7716 [id=f61735ea21334d049d389d7376ac3fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.609064 746 tablet_service.cc:1467] Processing CreateTablet for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a (DEFAULT_TABLE table=default.Table_61bd6e7ab702444ba0919bfb8efb04f0 [id=b8934690b09642ff81dcd01b665b6565]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.610899 741 tablet_service.cc:1467] Processing CreateTablet for tablet f3280e8ba6db4446aa3e3bfe2836abfa (DEFAULT_TABLE table=default.Table_5da68f9f7cdb439daae3b29bc1e40438 [id=3e382235d0e34eb6aad49f9c52df8603]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.612041 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3280e8ba6db4446aa3e3bfe2836abfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.612195 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_2e2abefaccf54e469bacf8126edad0c4 [id=855145809143431d81790dd36147a77b]). Expected version 2 got 0
I20250115 14:52:00.613417 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03eb225fad7459a955a3ec5e0002be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.613818 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 759a4ebe7b0444128ddd31a78aad463e
I20250115 14:52:00.617055 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b6a520bf69264c44b1da30c0ffb41cc0" } modify_external_catalogs: true
I20250115 14:52:00.617329 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b6a520bf69264c44b1da30c0ffb41cc0" } modify_external_catalogs: true
W20250115 14:52:00.621263 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 353476us
I20250115 14:52:00.621310 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89ec0a5d68408c897f43ab39df62b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.621750 892 tablet_service.cc:1467] Processing CreateTablet for tablet d6f2bc62621d402cacfcbe2eb74173f9 (DEFAULT_TABLE table=default.Table_62b16563153946409ee7c59069520182 [id=3249c65f52624113be96f7ef7b828854]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.621953 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.622072 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 4239747374aa4d6d866961430786677c (table default.Table_8e6652dab3bc46b7b4f1b66415398217 [id=8941c6c27037475990cf318614854a27]) successfully deleted
I20250115 14:52:00.622314 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 759a4ebe7b0444128ddd31a78aad463e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.622931 1166 tablet_replica.cc:331] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.624279 1168 tablet_bootstrap.cc:492] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:00.628764 878 tablet_service.cc:1467] Processing CreateTablet for tablet 1218f0bdb1e14bd78d76c6a06805f38c (DEFAULT_TABLE table=default.Table_272f9650cb3142f781f1d4d7c4769dfe [id=2fb684a787e4420aa9218b369eba5cf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.629142 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f541e916d264738842cd965c7a2a5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.630142 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6978223c1984a38bec5f2d06a9cf457. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.631842 1166 raft_consensus.cc:2238] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.632328 1166 raft_consensus.cc:2267] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.635205 1166 ts_tablet_manager.cc:1905] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.637878 1164 tablet_bootstrap.cc:654] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.641791 888 tablet_service.cc:1467] Processing CreateTablet for tablet 4adf1353bd554d8294a7f570931095d2 (DEFAULT_TABLE table=default.Table_0e5d68124c7a4a0dbde272f4fb012cea [id=1c9959ef09ed4988baf48112f0280303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.655021 1839 raft_consensus.cc:491] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.655444 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1839edd0d49b467d8bb31b3001ff89eb" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:00.669266 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:52:00.669919 1166 ts_tablet_manager.cc:1918] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.670064 1166 log.cc:1198] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e50462626f914fb1b0f440c48f30e72d
I20250115 14:52:00.670423 1166 ts_tablet_manager.cc:1939] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.671070 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e50462626f914fb1b0f440c48f30e72d (table default.Table_6dfc58433fe64bf48f2215d6caab2368 [id=c049c755539f4a29a3d34b7b89f20daf]) successfully deleted
I20250115 14:52:00.672133 1166 tablet_replica.cc:331] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.672366 1166 raft_consensus.cc:2238] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.672546 1166 raft_consensus.cc:2267] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.673472 1164 tablet_bootstrap.cc:492] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.673607 1164 ts_tablet_manager.cc:1397] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.000s
I20250115 14:52:00.674340 1164 raft_consensus.cc:357] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.674537 1164 raft_consensus.cc:383] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.674623 1164 raft_consensus.cc:738] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.674840 1164 consensus_queue.cc:260] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.678027 1164 ts_tablet_manager.cc:1428] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250115 14:52:00.679714 1164 tablet_bootstrap.cc:492] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.680738 750 tablet_service.cc:1514] Processing DeleteTablet for tablet f50edca843e54f4b8e803d4d180b736d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:53 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:00.681231 1855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250115 14:52:00.681730 1854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:52:00.687127 1164 tablet_bootstrap.cc:654] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.670284 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:52:00.690577 1808 server_base.cc:1034] running on GCE node
I20250115 14:52:00.694926 1166 ts_tablet_manager.cc:1905] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.697547 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 4adf1353bd554d8294a7f570931095d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.699379 1164 tablet_bootstrap.cc:492] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.699672 1164 ts_tablet_manager.cc:1397] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20250115 14:52:00.700103 746 tablet_service.cc:1467] Processing CreateTablet for tablet a9a8cd2e6f3a465181ec781cc92bc2f6 (DEFAULT_TABLE table=default.Table_68042c9611eb4780a797c5bb2aec65e9 [id=69cb20c803d443d5a73b63d31e11bd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.700103 745 tablet_service.cc:1467] Processing CreateTablet for tablet 02ce16fea4f9469c9fc06fa24f6554be (DEFAULT_TABLE table=default.Table_a2eb9b62487c4908880b44b9915e4cf7 [id=d15ba63620364fd18f1751a535cc3235]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.700557 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ce16fea4f9469c9fc06fa24f6554be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.700580 752 tablet_service.cc:1467] Processing CreateTablet for tablet f60e68b411364ada9b89879012d862ea (DEFAULT_TABLE table=default.Table_f8719599d09f457681ab1c5bbf617baa [id=ce85b2d5cbb3453e8237d9404cb9f94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.700932 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f60e68b411364ada9b89879012d862ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.704548 739 tablet_service.cc:1467] Processing CreateTablet for tablet d0f3f6de654d4765b1d7a381f93b0c43 (DEFAULT_TABLE table=default.Table_707a274426cd4d37a8ee193d73a0e171 [id=e7e8720c2f454d0e842cae5c2d32e5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.704952 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f3f6de654d4765b1d7a381f93b0c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.705111 743 tablet_service.cc:1467] Processing CreateTablet for tablet 57aecd22f97c41618f90d14c2f53a7c9 (DEFAULT_TABLE table=default.Table_9118acb2bd5e43e9b5d4b0d2379fadfb [id=492ed43384bd463dbfa14c65b58044e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.705447 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57aecd22f97c41618f90d14c2f53a7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.708575 751 tablet_service.cc:1467] Processing CreateTablet for tablet c6cb8dfa7c42480bbd139d5acf059a57 (DEFAULT_TABLE table=default.Table_dd6e49ce9dc4402b88b42b9ed72e08eb [id=630335788e2743a690d4879739aad240]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.708912 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6cb8dfa7c42480bbd139d5acf059a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.711129 1010 tablet_service.cc:1467] Processing CreateTablet for tablet ed777d7496514ec5986a7e6f53d532b1 (DEFAULT_TABLE table=default.Table_56da29ad1fe24546bc5c6a4a17a774e7 [id=1b1cf9caa0194880af26c65d6f7bcbf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.713250 750 tablet_service.cc:1467] Processing CreateTablet for tablet e180e37a1d5d46d78380323662b80f85 (DEFAULT_TABLE table=default.Table_ae2b381e4eb84e6788f5d29ffc7a72f7 [id=a2079a8b5634463b83a77df10fd713a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.713558 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e180e37a1d5d46d78380323662b80f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.716538 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9a8cd2e6f3a465181ec781cc92bc2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.718909 1164 raft_consensus.cc:357] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.722344 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8a7055866139444681c82d83d9af5978" } new_table_name: "default.Table_d8d3e6a881ce475a8f1319238ab8c610" modify_external_catalogs: true
I20250115 14:52:00.723780 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a8abdc319d2c413ba113c127fbfd5971" } new_table_name: "default.Table_609197916c9b4c81aebdaf70e78cc436" modify_external_catalogs: true
I20250115 14:52:00.727370 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3f8db86f4b5b4f1b88804f296cc86c3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.727814 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f8db86f4b5b4f1b88804f296cc86c3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.729144 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_33f4705d7a1b474d82734fe4a80c0424" } modify_external_catalogs: true
I20250115 14:52:00.729350 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_33f4705d7a1b474d82734fe4a80c0424" } modify_external_catalogs: true
I20250115 14:52:00.730510 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dae457edf44a40a69a40030ae24a26ff" } modify_external_catalogs: true
I20250115 14:52:00.730688 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dae457edf44a40a69a40030ae24a26ff" } modify_external_catalogs: true
I20250115 14:52:00.719794 1164 raft_consensus.cc:383] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.732681 1164 raft_consensus.cc:738] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.733728 741 tablet_service.cc:1467] Processing CreateTablet for tablet 9494c0c57be8499a883888d067c7db77 (DEFAULT_TABLE table=default.Table_6b196b51d11442e5be9d5a3a2f6f3484 [id=d6e71eafa9aa42d2bf53bf5c4bb82fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.733812 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_095514432af8489eab6e04a8b9bf1f53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:00.734164 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9494c0c57be8499a883888d067c7db77. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:00.734278 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_095514432af8489eab6e04a8b9bf1f53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:00.734665 1857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:52:00.736536 1808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 14:52:00.737157 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f9a5e1c80cc147cf861197fac6765b01" } new_table_name: "default.Table_a1dc2d81847c4f31bc83b06081b8274f" modify_external_catalogs: true
I20250115 14:52:00.740404 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83097c5bfe364418a213236435edc40c" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:00.742805 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 0cae4c9228c640e495507e8f7ad9f544 (DEFAULT_TABLE table=default.Table_dae457edf44a40a69a40030ae24a26ff [id=0dd965b7347c439d9abcc56446730332]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.743451 1166 ts_tablet_manager.cc:1918] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.743438 749 tablet_service.cc:1467] Processing CreateTablet for tablet 342bf8d4dc4244f982b1b63aaae0fb45 (DEFAULT_TABLE table=default.Table_1ee3b83d4fee4651b3979b44d4903db6 [id=1822bed451a74f2dbd54da029abd4384]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.743618 1166 log.cc:1198] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b25dce523e724066b203e6111cad8111
I20250115 14:52:00.744565 1164 consensus_queue.cc:260] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.745385 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 342bf8d4dc4244f982b1b63aaae0fb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.745947 1164 ts_tablet_manager.cc:1428] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.046s user 0.001s sys 0.003s
I20250115 14:52:00.747968 739 tablet_service.cc:1467] Processing CreateTablet for tablet f400880835ab4efdb2af0be5553d20dc (DEFAULT_TABLE table=default.Table_38b74aa452d14ca5b22b7259f85af561 [id=ce47660a249c41c68f6b6b2c2c9cd485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.749002 1166 ts_tablet_manager.cc:1939] T b25dce523e724066b203e6111cad8111 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.749320 1025 tablet_service.cc:1467] Processing CreateTablet for tablet abefc717e0eb41b18725ef0c73154fdd (DEFAULT_TABLE table=default.Table_014f98e470c94ba4b3446630153a9e75 [id=3be2f53f8e374ee6aa37100f87b50b1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.750106 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2d55a4e32448fbb98f704dbf1657b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.750396 1166 tablet_replica.cc:331] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.750170 1164 tablet_bootstrap.cc:492] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.751375 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de6f6f014074c3d8775cb5e4121ccb0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:00.751791 1166 raft_consensus.cc:2238] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.752568 1166 raft_consensus.cc:2267] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.753484 1166 ts_tablet_manager.cc:1905] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.757084 1750 raft_consensus.cc:491] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.757226 1750 raft_consensus.cc:513] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:00.758848 1732 raft_consensus.cc:513] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.759778 1732 leader_election.cc:290] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:00.759882 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c984613c7d45d5b7c513cd900c9558" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:00.760334 1750 leader_election.cc:290] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:00.760417 984 leader_election.cc:343] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:00.764519 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cae4c9228c640e495507e8f7ad9f544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.765259 1048 raft_consensus.cc:2463] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:00.765745 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed777d7496514ec5986a7e6f53d532b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.766767 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abefc717e0eb41b18725ef0c73154fdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.769034 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 72a97d102c174bc3a5e3a3c5762033b3 (DEFAULT_TABLE table=default.Table_0a8e879b253d48d3b02055cb2bd9663e [id=e07c949059424311bb6faf43de5dd443]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.769567 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b25dce523e724066b203e6111cad8111 (table default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]) successfully deleted
I20250115 14:52:00.770704 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f89ec0a5d68408c897f43ab39df62b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.777451 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7309ecb11d844848638c042c9bbe5f1" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:52:00.783056 710 leader_election.cc:343] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: e50462626f914fb1b0f440c48f30e72d
I20250115 14:52:00.783185 710 leader_election.cc:304] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:52:00.783392 710 raft_consensus.cc:2681] T e50462626f914fb1b0f440c48f30e72d P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:52:00.783846 1014 tablet_service.cc:1467] Processing CreateTablet for tablet e6978223c1984a38bec5f2d06a9cf457 (DEFAULT_TABLE table=default.Table_f9a5e1c80cc147cf861197fac6765b01 [id=e03ea31f74914ceba6e5a36ba1b30dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.784014 710 leader_election.cc:304] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:52:00.784406 710 leader_election.cc:343] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 50ca710cea154be5b0f57ace4af4168b
I20250115 14:52:00.784516 710 leader_election.cc:304] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:52:00.784689 710 raft_consensus.cc:2681] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:52:00.784929 1750 raft_consensus.cc:2743] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:00.785120 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a97d102c174bc3a5e3a3c5762033b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.781821 891 tablet_service.cc:1467] Processing CreateTablet for tablet ba33eb3f5a40450db5cf115fbe8a17e7 (DEFAULT_TABLE table=default.Table_a584af21c70545b8a2d3690c20422f1f [id=f8c4ca50220a4dbf9b9480213d836e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.781337 1839 raft_consensus.cc:513] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.786278 1835 raft_consensus.cc:491] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.786412 1835 raft_consensus.cc:513] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.786587 1844 raft_consensus.cc:491] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:00.786688 1844 raft_consensus.cc:513] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.787473 847 leader_election.cc:304] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:00.788175 1843 raft_consensus.cc:2798] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:00.788259 1843 raft_consensus.cc:491] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:00.788384 1843 raft_consensus.cc:3054] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:00.788511 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7309ecb11d844848638c042c9bbe5f1" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:00.789752 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1218f0bdb1e14bd78d76c6a06805f38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.792724 1162 tablet_bootstrap.cc:492] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:00.792871 1162 ts_tablet_manager.cc:1397] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.679s user 0.006s sys 0.010s
I20250115 14:52:00.794167 710 leader_election.cc:304] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:00.794499 1750 raft_consensus.cc:2743] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:00.795166 1838 consensus_queue.cc:1035] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20250115 14:52:00.799141 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c984613c7d45d5b7c513cd900c9558" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:00.799464 1162 raft_consensus.cc:357] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.799651 1162 raft_consensus.cc:383] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.799820 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f400880835ab4efdb2af0be5553d20dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.800719 1162 raft_consensus.cc:738] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.800956 1162 consensus_queue.cc:260] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.801509 1162 ts_tablet_manager.cc:1428] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250115 14:52:00.801792 1162 tablet_bootstrap.cc:492] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:00.802321 847 consensus_peers.cc:487] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5703fc5995184298bf72f9e43efb41f5. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:00.805018 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6f2bc62621d402cacfcbe2eb74173f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.805414 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cae4c9228c640e495507e8f7ad9f544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.805780 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83097c5bfe364418a213236435edc40c" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:00.805856 1164 tablet_bootstrap.cc:654] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.806085 908 raft_consensus.cc:3054] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:00.806358 879 tablet_service.cc:1467] Processing CreateTablet for tablet f3280e8ba6db4446aa3e3bfe2836abfa (DEFAULT_TABLE table=default.Table_5da68f9f7cdb439daae3b29bc1e40438 [id=3e382235d0e34eb6aad49f9c52df8603]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.806715 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3280e8ba6db4446aa3e3bfe2836abfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.807775 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4adf1353bd554d8294a7f570931095d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.808583 710 leader_election.cc:304] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:00.808866 1750 raft_consensus.cc:2743] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:00.803028 1011 tablet_service.cc:1467] Processing CreateTablet for tablet d1259256b7744e739581125611abbe67 (DEFAULT_TABLE table=default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.809360 1166 ts_tablet_manager.cc:1918] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.809497 1166 log.cc:1198] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/50ca710cea154be5b0f57ace4af4168b
I20250115 14:52:00.809486 877 tablet_service.cc:1467] Processing CreateTablet for tablet e910c387e6a24caf95e14b57c8b83e6f (DEFAULT_TABLE table=default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.809820 1166 ts_tablet_manager.cc:1939] T 50ca710cea154be5b0f57ace4af4168b P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.810245 1168 tablet_bootstrap.cc:654] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.810415 885 tablet_service.cc:1467] Processing CreateTablet for tablet cc9ce226d3ba44d6908257bbd0df3a12 (DEFAULT_TABLE table=default.Table_2c94d0bb9dd54c5195b1fcbf7cec15fd [id=ceccf8508c5047c2b7a63788dd81a66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.810667 1166 tablet_replica.cc:331] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.810871 1166 raft_consensus.cc:2238] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.810946 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 50ca710cea154be5b0f57ace4af4168b (table default.Table_d809e208790b4c1a911708b393be6a3f [id=d45a9a753a874c41a842f25450ab10d2]) successfully deleted
I20250115 14:52:00.811033 1166 raft_consensus.cc:2267] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.814039 1839 leader_election.cc:290] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:00.814556 1166 ts_tablet_manager.cc:1905] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.816053 1835 leader_election.cc:290] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:00.817888 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba33eb3f5a40450db5cf115fbe8a17e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.819000 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e910c387e6a24caf95e14b57c8b83e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.820003 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9ce226d3ba44d6908257bbd0df3a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.816936 1844 leader_election.cc:290] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:00.824920 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6978223c1984a38bec5f2d06a9cf457. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.833352 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72a97d102c174bc3a5e3a3c5762033b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.836999 1751 ts_tablet_manager.cc:1918] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.837539 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 2560d42c15c64b4c9c0cfe95706b4d9e (DEFAULT_TABLE table=default.Table_99a5f2ca224f459dbf6bd4d4ec255457 [id=92ab45b4bf154e789e3a9b33c77d38c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.846835 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 1218f0bdb1e14bd78d76c6a06805f38c (DEFAULT_TABLE table=default.Table_272f9650cb3142f781f1d4d7c4769dfe [id=2fb684a787e4420aa9218b369eba5cf2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:00.849273 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 580742eb5dac4dba9995809890e0e315 consensus state has no leader
I20250115 14:52:00.849555 1166 ts_tablet_manager.cc:1918] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.849670 1166 log.cc:1198] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b9ce445d334842c8a6bacad9f693feda
I20250115 14:52:00.849999 1166 ts_tablet_manager.cc:1939] T b9ce445d334842c8a6bacad9f693feda P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.851226 1166 tablet_replica.cc:331] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.851426 1166 raft_consensus.cc:2238] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.851608 1166 raft_consensus.cc:2267] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:00.851945 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b9ce445d334842c8a6bacad9f693feda (table default.Table_ff3a0751d563485db14fc5a44116f2ac [id=4334f62c5f0a43119d21690c19315404]) successfully deleted
I20250115 14:52:00.852149 1166 ts_tablet_manager.cc:1905] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.852929 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0cae4c9228c640e495507e8f7ad9f544
I20250115 14:52:00.854099 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fc4f396af4f54518b0232f10057dd159" } new_table_name: "default.Table_4f00ef2ef63d4f77929a245aa9bff035" modify_external_catalogs: true
I20250115 14:52:00.854300 749 tablet_service.cc:1514] Processing DeleteTablet for tablet f47a8acc3bec46b78725dcc3e675ea74 with delete_type TABLET_DATA_DELETED (replaced by tablet ea8ab6a34caa486ab929af2cac3ce957) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.855448 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6308f68e41554470b7d1321c60d4eee9
I20250115 14:52:00.856173 1164 tablet_bootstrap.cc:492] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.856328 1164 ts_tablet_manager.cc:1397] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.012s
I20250115 14:52:00.857121 1164 raft_consensus.cc:357] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.857313 1164 raft_consensus.cc:383] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.857391 1164 raft_consensus.cc:738] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.857606 1164 consensus_queue.cc:260] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.858132 1164 ts_tablet_manager.cc:1428] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:00.860878 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ef39f0505cbe4de9b46458d25eafb9e1" } modify_external_catalogs: true
I20250115 14:52:00.861047 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ef39f0505cbe4de9b46458d25eafb9e1" } modify_external_catalogs: true
I20250115 14:52:00.861411 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0437d3856b9a4dccbdf5e1da42cbe92f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.861940 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0437d3856b9a4dccbdf5e1da42cbe92f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.862505 1164 tablet_bootstrap.cc:492] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.863622 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ec781cbe91bf4224b2b7c28a98ccac67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.863987 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6978223c1984a38bec5f2d06a9cf457 consensus state has no leader
W20250115 14:52:00.864131 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec781cbe91bf4224b2b7c28a98ccac67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.864446 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8234c8fb043a45b495995ce2bcd14cc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.864912 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8234c8fb043a45b495995ce2bcd14cc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.866425 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_70c7c9d33ea742618f8a32464ed92f8e" } new_table_name: "default.Table_4b79de2dd7fe493f84a77ae3c477e2c8" modify_external_catalogs: true
I20250115 14:52:00.867396 1164 tablet_bootstrap.cc:654] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:00.867434 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4f89ec0a5d68408c897f43ab39df62b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f89ec0a5d68408c897f43ab39df62b1 already in progress: creating tablet
I20250115 14:52:00.867466 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9165664ce60447b3a00c3ac8006c7ebe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.867720 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4adf1353bd554d8294a7f570931095d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4adf1353bd554d8294a7f570931095d2 already in progress: creating tablet
W20250115 14:52:00.867934 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9165664ce60447b3a00c3ac8006c7ebe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.868872 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_116f78c6117c4cf4b8db61d75ae62013"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:00.869936 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b5177265ba974ad6a381adb1e8c49589
W20250115 14:52:00.870178 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_116f78c6117c4cf4b8db61d75ae62013 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.871337 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cae4c9228c640e495507e8f7ad9f544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.871522 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6308f68e41554470b7d1321c60d4eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:00.875651 749 tablet_service.cc:1514] Processing DeleteTablet for tablet b5177265ba974ad6a381adb1e8c49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:00.876147 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e49dc97eea474252a5e0ef4af6e015a9 consensus state has no leader
I20250115 14:52:00.876587 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3d9f1f67737c458aa1d0004f77e93808"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:00.876818 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3657238a564a4fec99af530ff70671af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.877030 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d9f1f67737c458aa1d0004f77e93808 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:00.877255 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3657238a564a4fec99af530ff70671af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.886817 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1259256b7744e739581125611abbe67. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.890429 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7c155cee1f6e49a99d8ab38956a737e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:00.891428 847 leader_election.cc:343] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 1f1df9dc743342b4b7bdb1a8d3fcd5b0
W20250115 14:52:00.891794 847 leader_election.cc:343] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 75788f8eb36b446e927197f624fbfdf5
W20250115 14:52:00.892832 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c155cee1f6e49a99d8ab38956a737e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:00.894723 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1218f0bdb1e14bd78d76c6a06805f38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.896835 1016 tablet_service.cc:1467] Processing CreateTablet for tablet f3280e8ba6db4446aa3e3bfe2836abfa (DEFAULT_TABLE table=default.Table_5da68f9f7cdb439daae3b29bc1e40438 [id=3e382235d0e34eb6aad49f9c52df8603]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.898190 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 4adf1353bd554d8294a7f570931095d2 (DEFAULT_TABLE table=default.Table_0e5d68124c7a4a0dbde272f4fb012cea [id=1c9959ef09ed4988baf48112f0280303]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.899272 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 3f541e916d264738842cd965c7a2a5ec (DEFAULT_TABLE table=default.Table_33a2bbb1ba7d426caf1eee40268cec01 [id=8d19600fb0354c028cafa03a1c87e5c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.878432 1166 ts_tablet_manager.cc:1918] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:00.900877 1166 log.cc:1198] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/4b90205ef15f470a99c013e7f2cf99c6
I20250115 14:52:00.901314 1166 ts_tablet_manager.cc:1939] T 4b90205ef15f470a99c013e7f2cf99c6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.902495 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 4b90205ef15f470a99c013e7f2cf99c6 (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]) successfully deleted
I20250115 14:52:00.902786 1166 tablet_replica.cc:331] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.902753 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2560d42c15c64b4c9c0cfe95706b4d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.902966 1166 raft_consensus.cc:2238] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:00.903074 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e034d7761a5247c79414b7112a1c97b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e034d7761a5247c79414b7112a1c97b2 already in progress: creating tablet
I20250115 14:52:00.903141 1166 raft_consensus.cc:2267] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:00.903523 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9145bbaf3e234f079e355454f3ce28cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9145bbaf3e234f079e355454f3ce28cb already in progress: creating tablet
W20250115 14:52:00.903777 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9f4475441e4a467a8d593b742d2a5b80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f4475441e4a467a8d593b742d2a5b80 already in progress: creating tablet
I20250115 14:52:00.908097 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d6f2bc62621d402cacfcbe2eb74173f9 (DEFAULT_TABLE table=default.Table_62b16563153946409ee7c59069520182 [id=3249c65f52624113be96f7ef7b828854]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.908439 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3280e8ba6db4446aa3e3bfe2836abfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.909345 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89ec0a5d68408c897f43ab39df62b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.912883 1751 log.cc:1198] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e7100e6a51ed4e178e86138f10d5ffc2
I20250115 14:52:00.914115 1166 ts_tablet_manager.cc:1905] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.921047 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:52:00.921120 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_33a2bbb1ba7d426caf1eee40268cec01" } modify_external_catalogs: true
I20250115 14:52:00.921295 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_33a2bbb1ba7d426caf1eee40268cec01" } modify_external_catalogs: true
I20250115 14:52:00.921766 749 tablet_service.cc:1514] Processing DeleteTablet for tablet a73bd0141ccc4ec4b052b18b9fb62bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:00.921795 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 consensus state has no leader
I20250115 14:52:00.922338 1844 consensus_queue.cc:1035] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20250115 14:52:00.923110 1166 ts_tablet_manager.cc:1918] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.923215 1166 log.cc:1198] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/7a9e2c053d624b6ba71b031a591475a4
I20250115 14:52:00.923537 1166 ts_tablet_manager.cc:1939] T 7a9e2c053d624b6ba71b031a591475a4 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.926359 908 raft_consensus.cc:2463] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
W20250115 14:52:00.928539 847 leader_election.cc:343] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: f847f92cd122423aa93f1afa701b0bbb
I20250115 14:52:00.930141 1021 tablet_service.cc:1467] Processing CreateTablet for tablet e910c387e6a24caf95e14b57c8b83e6f (DEFAULT_TABLE table=default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.928095 1865 tablet.cc:1721] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:00.934890 1015 tablet_service.cc:1467] Processing CreateTablet for tablet ba33eb3f5a40450db5cf115fbe8a17e7 (DEFAULT_TABLE table=default.Table_a584af21c70545b8a2d3690c20422f1f [id=f8c4ca50220a4dbf9b9480213d836e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.939813 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4475441e4a467a8d593b742d2a5b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.942766 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f541e916d264738842cd965c7a2a5ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.945739 1166 tablet_replica.cc:331] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.945719 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 7a9e2c053d624b6ba71b031a591475a4 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]) successfully deleted
I20250115 14:52:00.945964 1166 raft_consensus.cc:2238] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.946130 1166 raft_consensus.cc:2267] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:00.946148 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 759a4ebe7b0444128ddd31a78aad463e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 759a4ebe7b0444128ddd31a78aad463e already in progress: creating tablet
I20250115 14:52:00.949051 1166 ts_tablet_manager.cc:1905] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:00.949507 1162 tablet_bootstrap.cc:654] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:00.954566 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba33eb3f5a40450db5cf115fbe8a17e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.956312 1166 ts_tablet_manager.cc:1918] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:00.956442 1166 log.cc:1198] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f50edca843e54f4b8e803d4d180b736d
I20250115 14:52:00.956751 1166 ts_tablet_manager.cc:1939] T f50edca843e54f4b8e803d4d180b736d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:00.957619 1166 tablet_replica.cc:331] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:00.957792 1166 raft_consensus.cc:2238] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:00.957964 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f50edca843e54f4b8e803d4d180b736d (table default.Table_490c22b7496a4d7db6bc6955e246363b [id=93b72d2ab0bc4ca2bf675a110a5bf4f2]) successfully deleted
I20250115 14:52:00.957479 983 leader_election.cc:304] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:00.961215 1751 ts_tablet_manager.cc:1939] T e7100e6a51ed4e178e86138f10d5ffc2 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:00.964425 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 4adf1353bd554d8294a7f570931095d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.957320 1022 tablet_service.cc:1467] Processing CreateTablet for tablet cc9ce226d3ba44d6908257bbd0df3a12 (DEFAULT_TABLE table=default.Table_2c94d0bb9dd54c5195b1fcbf7cec15fd [id=ceccf8508c5047c2b7a63788dd81a66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.968832 881 tablet_service.cc:1514] Processing DeleteTablet for tablet e034d7761a5247c79414b7112a1c97b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.968914 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e910c387e6a24caf95e14b57c8b83e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.970777 1728 raft_consensus.cc:2798] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:00.972787 1728 raft_consensus.cc:695] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:00.973130 1728 consensus_queue.cc:237] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.973505 1808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 14:52:00.974974 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6f2bc62621d402cacfcbe2eb74173f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.975102 749 tablet_service.cc:1467] Processing CreateTablet for tablet 11fddbd9ba14443a91865387e1d1c238 (DEFAULT_TABLE table=default.Table_3f8db86f4b5b4f1b88804f296cc86c3f [id=311b894955f4455aa727f31c46879bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.975476 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fddbd9ba14443a91865387e1d1c238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.980027 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a79f880415a4b16aabdbb7cd989efdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.980931 1808 hybrid_clock.cc:648] HybridClock initialized: now 1736952720980792 us; error 96 us; skew 500 ppm
I20250115 14:52:00.981375 1164 tablet_bootstrap.cc:492] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:00.981356 1808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250115 14:52:00.981535 1164 ts_tablet_manager.cc:1397] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.119s user 0.001s sys 0.021s
I20250115 14:52:00.982858 1164 raft_consensus.cc:357] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.983099 1164 raft_consensus.cc:383] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:00.983171 1164 raft_consensus.cc:738] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:00.983399 1164 consensus_queue.cc:260] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:00.983914 1164 ts_tablet_manager.cc:1428] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:00.984227 1164 tablet_bootstrap.cc:492] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:00.987977 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:52:00.989394 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet e034d7761a5247c79414b7112a1c97b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.991047 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:52:00.991645 875 tablet_service.cc:1467] Processing CreateTablet for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c (DEFAULT_TABLE table=default.Table_c9a1822ec7ae455ba4b2d5a1181d3eb1 [id=b085d42488df4fee9c6da3f98b0a163a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.991684 1866 tablet.cc:1721] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:00.992910 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9145bbaf3e234f079e355454f3ce28cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.993322 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4475441e4a467a8d593b742d2a5b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:00.994010 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c (DEFAULT_TABLE table=default.Table_c9a1822ec7ae455ba4b2d5a1181d3eb1 [id=b085d42488df4fee9c6da3f98b0a163a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.995251 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed777d7496514ec5986a7e6f53d532b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.995553 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 9145bbaf3e234f079e355454f3ce28cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:00.997743 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 6203efb862ea4962a6d09f3e4306a92f (DEFAULT_TABLE table=default.Table_d18e82c7d46c4394b6037413cd35cef6 [id=530bcb9252fa421b8df0f5932117a36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:00.998698 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 473) took 1148 ms. Trace:
I20250115 14:52:00.998898 1751 rpcz_store.cc:276] 0115 14:51:59.849691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.849741 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:00.115946 (+266205us) ts_tablet_manager.cc:1145] Deleting tablet e7100e6a51ed4e178e86138f10d5ffc2
0115 14:52:00.115955 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:00.278391 (+162436us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:00.836869 (+558478us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:00.998619 (+161750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":266023}
I20250115 14:52:00.999796 1751 tablet_replica.cc:331] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:01.003976 1750 raft_consensus.cc:491] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.004140 1750 raft_consensus.cc:513] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.004823 1750 leader_election.cc:290] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.005292 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.006673 710 leader_election.cc:343] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: a73bd0141ccc4ec4b052b18b9fb62bb7
I20250115 14:52:01.007349 881 tablet_service.cc:1467] Processing CreateTablet for tablet 6203efb862ea4962a6d09f3e4306a92f (DEFAULT_TABLE table=default.Table_d18e82c7d46c4394b6037413cd35cef6 [id=530bcb9252fa421b8df0f5932117a36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.008466 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd (DEFAULT_TABLE table=default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.007040 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd (DEFAULT_TABLE table=default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:01.011194 710 leader_election.cc:343] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: a73bd0141ccc4ec4b052b18b9fb62bb7
I20250115 14:52:01.011309 710 leader_election.cc:304] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:01.011662 1750 raft_consensus.cc:2743] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:01.012246 848 consensus_peers.cc:487] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:01.016894 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f1df9dc743342b4b7bdb1a8d3fcd5b0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.017004 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75788f8eb36b446e927197f624fbfdf5" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.017277 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f847f92cd122423aa93f1afa701b0bbb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.012969 1843 raft_consensus.cc:513] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.020658 1164 tablet_bootstrap.cc:654] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.026474 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9ce226d3ba44d6908257bbd0df3a12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:00.966205 1166 raft_consensus.cc:2267] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.030689 1843 leader_election.cc:290] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:01.031342 848 consensus_peers.cc:487] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:01.032428 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.032461 1012 tablet_service.cc:1467] Processing CreateTablet for tablet d03eb225fad7459a955a3ec5e0002be1 (DEFAULT_TABLE table=default.Table_9daeb4cb953b43a6bb9dc34b756d7716 [id=f61735ea21334d049d389d7376ac3fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.032788 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03eb225fad7459a955a3ec5e0002be1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.034480 848 leader_election.cc:343] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.034600 848 leader_election.cc:304] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:01.034979 848 leader_election.cc:343] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.035102 848 leader_election.cc:304] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:01.035483 848 leader_election.cc:343] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.035580 848 leader_election.cc:304] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.036144 1838 raft_consensus.cc:2743] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.036324 1838 raft_consensus.cc:2743] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.036489 1838 raft_consensus.cc:2743] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.037734 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4adf1353bd554d8294a7f570931095d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.038570 1751 raft_consensus.cc:2238] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.039670 1698 raft_consensus.cc:491] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.045707 1540 raft_consensus.cc:491] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.051272 1698 raft_consensus.cc:513] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.051555 1162 tablet_bootstrap.cc:492] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.052588 1162 ts_tablet_manager.cc:1397] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.250s user 0.007s sys 0.003s
I20250115 14:52:01.052805 1865 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:01.054322 1167 ts_tablet_manager.cc:1918] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.054877 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6203efb862ea4962a6d09f3e4306a92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.055830 880 tablet_service.cc:1467] Processing CreateTablet for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a (DEFAULT_TABLE table=default.Table_61bd6e7ab702444ba0919bfb8efb04f0 [id=b8934690b09642ff81dcd01b665b6565]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.055900 1166 ts_tablet_manager.cc:1905] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.056041 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9165664ce60447b3a00c3ac8006c7ebe" } new_table_name: "default.Table_d495ddf9033f421e9d6cc7c06d08c8f2" modify_external_catalogs: true
I20250115 14:52:01.056434 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_05f796ba50b14c1f973f89c85c133164" } new_table_name: "default.Table_4ab29505c6634887aa99118862eccf5d" modify_external_catalogs: true
I20250115 14:52:01.057053 775 raft_consensus.cc:3054] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:01.057368 776 raft_consensus.cc:3054] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:01.058858 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de6f6f014074c3d8775cb5e4121ccb0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:01.059322 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97c2f443ab040f9bc1924f45379165e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.059757 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e9bb748dd0854fe6ab5fb3dad43da484"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.060345 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9bb748dd0854fe6ab5fb3dad43da484 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.062968 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.063014 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e49dc97eea474252a5e0ef4af6e015a9 (table default.Table_4f00ef2ef63d4f77929a245aa9bff035 [id=8502b5fe264f4fb5a89bd4a91949c90b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e49dc97eea474252a5e0ef4af6e015a9 consensus state has no leader
I20250115 14:52:01.065820 884 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c81f47cd6e4a308cd62124db38d42f with delete_type TABLET_DATA_DELETED (replaced by tablet 580742eb5dac4dba9995809890e0e315) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.066182 884 tablet_service.cc:1467] Processing CreateTablet for tablet a9a8cd2e6f3a465181ec781cc92bc2f6 (DEFAULT_TABLE table=default.Table_68042c9611eb4780a797c5bb2aec65e9 [id=69cb20c803d443d5a73b63d31e11bd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.066550 1167 log.cc:1198] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/0291acf45d75421b8b1c6311560e7f0c
W20250115 14:52:01.068370 984 consensus_peers.cc:487] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4239747374aa4d6d866961430786677c. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:01.068729 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de6f6f014074c3d8775cb5e4121ccb0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:01.066529 876 tablet_service.cc:1467] Processing CreateTablet for tablet d03eb225fad7459a955a3ec5e0002be1 (DEFAULT_TABLE table=default.Table_9daeb4cb953b43a6bb9dc34b756d7716 [id=f61735ea21334d049d389d7376ac3fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.069684 1164 tablet_bootstrap.cc:492] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:01.069829 1164 ts_tablet_manager.cc:1397] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.086s user 0.005s sys 0.001s
I20250115 14:52:01.070884 1164 raft_consensus.cc:357] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.071076 1164 raft_consensus.cc:383] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.071224 1164 raft_consensus.cc:738] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.071451 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0437d3856b9a4dccbdf5e1da42cbe92f" } new_table_name: "default.Table_76ced15792e04bd0890267d54f928393" modify_external_catalogs: true
I20250115 14:52:01.071435 1164 consensus_queue.cc:260] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.071558 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9f7b9d2bd72741fbb3132278c8a77d1c" } modify_external_catalogs: true
I20250115 14:52:01.071724 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9f7b9d2bd72741fbb3132278c8a77d1c" } modify_external_catalogs: true
I20250115 14:52:01.072356 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d050f7c481a549248abc7c8beb08de6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.072824 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d050f7c481a549248abc7c8beb08de6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.073715 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_2e2abefaccf54e469bacf8126edad0c4 [id=855145809143431d81790dd36147a77b]). Expected version 2 got 0
I20250115 14:52:01.073355 1164 ts_tablet_manager.cc:1428] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:01.074520 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_06a952903c7b4dcc8088b8c9ff5a2ac5" } modify_external_catalogs: true
I20250115 14:52:01.074707 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_06a952903c7b4dcc8088b8c9ff5a2ac5" } modify_external_catalogs: true
I20250115 14:52:01.073843 1166 ts_tablet_manager.cc:1918] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.075266 1698 leader_election.cc:290] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.076201 1166 log.cc:1198] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f47a8acc3bec46b78725dcc3e675ea74
I20250115 14:52:01.068970 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 759a4ebe7b0444128ddd31a78aad463e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.077198 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cb0308c238d7479dbcc041fe3d6e7eff" } modify_external_catalogs: true
I20250115 14:52:01.077342 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cb0308c238d7479dbcc041fe3d6e7eff" } modify_external_catalogs: true
I20250115 14:52:01.077577 1164 tablet_bootstrap.cc:492] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:01.080258 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6203efb862ea4962a6d09f3e4306a92f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.081303 1166 ts_tablet_manager.cc:1939] T f47a8acc3bec46b78725dcc3e675ea74 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:52:01.081916 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 282d988bc41443c381bb0ba69b618f5d consensus state has no leader
I20250115 14:52:01.069708 1162 raft_consensus.cc:357] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.085007 886 tablet_service.cc:1467] Processing CreateTablet for tablet e180e37a1d5d46d78380323662b80f85 (DEFAULT_TABLE table=default.Table_ae2b381e4eb84e6788f5d29ffc7a72f7 [id=a2079a8b5634463b83a77df10fd713a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.086089 879 tablet_service.cc:1467] Processing CreateTablet for tablet 9494c0c57be8499a883888d067c7db77 (DEFAULT_TABLE table=default.Table_6b196b51d11442e5be9d5a3a2f6f3484 [id=d6e71eafa9aa42d2bf53bf5c4bb82fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.087389 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f47a8acc3bec46b78725dcc3e675ea74 (table default.Table_aa6a106222694bbfbe6c26b0f8e65396 [id=8725c85d30684db495ec7a20962fd75d]) successfully deleted
I20250115 14:52:01.087911 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d03eb225fad7459a955a3ec5e0002be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.088920 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97c2f443ab040f9bc1924f45379165e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:01.089143 1167 ts_tablet_manager.cc:1939] T 0291acf45d75421b8b1c6311560e7f0c P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:01.090377 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f129f5835a304494a797ad1eb37cf8fe" } new_table_name: "default.Table_96eab6cd20b842138b5acc7c8d0ac5d6" modify_external_catalogs: true
I20250115 14:52:01.091575 1808 webserver.cc:458] Webserver started at http://127.0.104.61:40195/ using document root <none> and password file <none>
I20250115 14:52:01.091686 1540 raft_consensus.cc:513] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.092284 1540 leader_election.cc:290] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.092744 1808 fs_manager.cc:362] Metadata directory not provided
I20250115 14:52:01.092911 1808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 14:52:01.092468 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 759a4ebe7b0444128ddd31a78aad463e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.095678 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e7100e6a51ed4e178e86138f10d5ffc2 (table default.Table_600b928b6de9443db6b3c73398af80b7 [id=fa886a7e8fcd43a096892c6435e13c29]) successfully deleted
I20250115 14:52:01.096251 1751 pending_rounds.cc:62] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:52:01.096355 1751 pending_rounds.cc:69] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558338227728384 op_type: NO_OP noop_request { }
I20250115 14:52:01.096485 1751 raft_consensus.cc:2883] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:01.096580 1751 raft_consensus.cc:2267] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:01.098415 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e6d67ddf01e34541bb795755e03c8249 has no consensus state
I20250115 14:52:01.099056 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a (DEFAULT_TABLE table=default.Table_61bd6e7ab702444ba0919bfb8efb04f0 [id=b8934690b09642ff81dcd01b665b6565]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:01.099265 984 leader_election.cc:343] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:01.099752 983 leader_election.cc:343] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.100106 983 leader_election.cc:304] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.101101 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:52:01.105291 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:52:01.105345 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.105976 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:52:01.107142 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4102b1f83e414d93a6a68315e36d16" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.107190 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:52:01.108625 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_2e2abefaccf54e469bacf8126edad0c4 [id=855145809143431d81790dd36147a77b]). Expected version 2 got 1
I20250115 14:52:01.112705 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 9494c0c57be8499a883888d067c7db77 (DEFAULT_TABLE table=default.Table_6b196b51d11442e5be9d5a3a2f6f3484 [id=d6e71eafa9aa42d2bf53bf5c4bb82fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.113777 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.113929 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4102b1f83e414d93a6a68315e36d16" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:01.114023 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.115432 984 leader_election.cc:343] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.115790 1540 raft_consensus.cc:2743] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:01.116545 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0cae4c9228c640e495507e8f7ad9f544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cae4c9228c640e495507e8f7ad9f544 already in progress: creating tablet
W20250115 14:52:01.117493 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6308f68e41554470b7d1321c60d4eee9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6308f68e41554470b7d1321c60d4eee9 already in progress: creating tablet
I20250115 14:52:01.119534 1808 fs_manager.cc:714] Time spent opening directory manager: real 0.024s user 0.009s sys 0.000s
I20250115 14:52:01.121062 1162 raft_consensus.cc:383] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.125646 1866 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:01.127943 891 tablet_service.cc:1467] Processing CreateTablet for tablet f60e68b411364ada9b89879012d862ea (DEFAULT_TABLE table=default.Table_f8719599d09f457681ab1c5bbf617baa [id=ce85b2d5cbb3453e8237d9404cb9f94e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:01.128599 1863 consensus_queue.cc:1159] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c238c037a1c74725a001e2b4a1ff4304" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250115 14:52:01.128839 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a9a8cd2e6f3a465181ec781cc92bc2f6 (DEFAULT_TABLE table=default.Table_68042c9611eb4780a797c5bb2aec65e9 [id=69cb20c803d443d5a73b63d31e11bd6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.129565 1844 raft_consensus.cc:491] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.130452 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9494c0c57be8499a883888d067c7db77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.130466 1162 raft_consensus.cc:738] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.131090 883 tablet_service.cc:1467] Processing CreateTablet for tablet d0f3f6de654d4765b1d7a381f93b0c43 (DEFAULT_TABLE table=default.Table_707a274426cd4d37a8ee193d73a0e171 [id=e7e8720c2f454d0e842cae5c2d32e5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.133191 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9a8cd2e6f3a465181ec781cc92bc2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.135615 1048 raft_consensus.cc:3054] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:01.136334 1024 tablet_service.cc:1467] Processing CreateTablet for tablet e180e37a1d5d46d78380323662b80f85 (DEFAULT_TABLE table=default.Table_ae2b381e4eb84e6788f5d29ffc7a72f7 [id=a2079a8b5634463b83a77df10fd713a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.138729 775 raft_consensus.cc:3054] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:01.135376 1844 raft_consensus.cc:513] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.139616 1168 tablet_bootstrap.cc:492] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:01.137100 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d0f3f6de654d4765b1d7a381f93b0c43 (DEFAULT_TABLE table=default.Table_707a274426cd4d37a8ee193d73a0e171 [id=e7e8720c2f454d0e842cae5c2d32e5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.146411 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 57aecd22f97c41618f90d14c2f53a7c9 (DEFAULT_TABLE table=default.Table_9118acb2bd5e43e9b5d4b0d2379fadfb [id=492ed43384bd463dbfa14c65b58044e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.146925 1164 tablet_bootstrap.cc:654] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.147064 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9494c0c57be8499a883888d067c7db77. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.150969 983 leader_election.cc:343] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.151182 983 leader_election.cc:304] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:01.153185 847 consensus_peers.cc:487] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:01.159081 1166 tablet_replica.cc:331] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:01.159296 1166 raft_consensus.cc:2238] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.159489 1166 raft_consensus.cc:2267] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.159664 1865 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43552 (request call id 482) took 1085 ms. Trace:
I20250115 14:52:01.160050 1162 consensus_queue.cc:260] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.162148 1166 ts_tablet_manager.cc:1905] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.163879 1751 ts_tablet_manager.cc:1905] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.165212 1018 tablet_service.cc:1467] Processing CreateTablet for tablet c6cb8dfa7c42480bbd139d5acf059a57 (DEFAULT_TABLE table=default.Table_dd6e49ce9dc4402b88b42b9ed72e08eb [id=630335788e2743a690d4879739aad240]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.165529 1732 raft_consensus.cc:2743] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.168642 1162 ts_tablet_manager.cc:1428] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.110s user 0.002s sys 0.000s
I20250115 14:52:01.169873 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 435) took 1656 ms. Trace:
I20250115 14:52:01.169996 1167 rpcz_store.cc:276] 0115 14:51:59.456656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.456703 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:00.339853 (+883150us) ts_tablet_manager.cc:1145] Deleting tablet 0291acf45d75421b8b1c6311560e7f0c
0115 14:52:00.339858 (+ 5us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.013639 (+673781us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.054223 (+ 40584us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.113095 (+ 58872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":882940}
I20250115 14:52:01.171921 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e180e37a1d5d46d78380323662b80f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.172320 749 tablet_service.cc:1467] Processing CreateTablet for tablet e6d67ddf01e34541bb795755e03c8249 (DEFAULT_TABLE table=default.Table_d495ddf9033f421e9d6cc7c06d08c8f2 [id=cf83ec5e8f3c4a0d92bdd962fbba9a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.172715 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6d67ddf01e34541bb795755e03c8249. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.172917 1167 tablet_replica.cc:331] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:01.173121 1167 raft_consensus.cc:2238] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.174774 1167 raft_consensus.cc:2267] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.175513 739 tablet_service.cc:1467] Processing CreateTablet for tablet 7718729182a340fe9b2dfea5a8321c1f (DEFAULT_TABLE table=default.Table_8234c8fb043a45b495995ce2bcd14cc7 [id=199476de8a584f9e9f6bcf826e83146b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.175872 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7718729182a340fe9b2dfea5a8321c1f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.177733 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b5177265ba974ad6a381adb1e8c49589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5177265ba974ad6a381adb1e8c49589 already in progress: creating tablet
I20250115 14:52:01.177906 741 tablet_service.cc:1467] Processing CreateTablet for tablet 5882252383ba46aba0ec053fab1fd383 (DEFAULT_TABLE table=default.Table_095514432af8489eab6e04a8b9bf1f53 [id=1532f36247a340cbba3c9b792d500c04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.178221 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5882252383ba46aba0ec053fab1fd383. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.171572 887 tablet_service.cc:1467] Processing CreateTablet for tablet 02ce16fea4f9469c9fc06fa24f6554be (DEFAULT_TABLE table=default.Table_a2eb9b62487c4908880b44b9915e4cf7 [id=d15ba63620364fd18f1751a535cc3235]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:01.171834 848 leader_election.cc:343] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.182889 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9a8cd2e6f3a465181ec781cc92bc2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.183841 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:52:01.184114 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_2e2abefaccf54e469bacf8126edad0c4 [id=855145809143431d81790dd36147a77b]). Expected version 2 got 0
I20250115 14:52:01.184896 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 02ce16fea4f9469c9fc06fa24f6554be (DEFAULT_TABLE table=default.Table_a2eb9b62487c4908880b44b9915e4cf7 [id=d15ba63620364fd18f1751a535cc3235]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.186055 1162 tablet_bootstrap.cc:492] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.182915 1016 tablet_service.cc:1467] Processing CreateTablet for tablet f60e68b411364ada9b89879012d862ea (DEFAULT_TABLE table=default.Table_f8719599d09f457681ab1c5bbf617baa [id=ce85b2d5cbb3453e8237d9404cb9f94e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.186400 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:52:01.187173 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f3220f35253d4aa087fd1d48bd8dc8b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:01.187441 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e180e37a1d5d46d78380323662b80f85. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.187680 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3220f35253d4aa087fd1d48bd8dc8b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.187954 889 tablet_service.cc:1467] Processing CreateTablet for tablet c6cb8dfa7c42480bbd139d5acf059a57 (DEFAULT_TABLE table=default.Table_dd6e49ce9dc4402b88b42b9ed72e08eb [id=630335788e2743a690d4879739aad240]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.189906 1844 leader_election.cc:290] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:01.191100 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c81f47cd6e4a308cd62124db38d42f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:01.191376 1048 raft_consensus.cc:2463] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:01.166594 1865 rpcz_store.cc:276] 0115 14:52:00.037529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.040397 (+ 2868us) service_pool.cc:225] Handling call
0115 14:52:01.123124 (+1082727us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0115 14:52:00.040833 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0115 14:52:00.040936 (+ 103us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0115 14:52:00.040945 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0115 14:52:00.040951 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0115 14:52:00.040986 (+ 35us) alter_schema_op.cc:128] START. Timestamp: 1736952720040952
0115 14:52:00.040989 (+ 3us) op_driver.cc:348] REPLICATION: starting
0115 14:52:00.041165 (+ 176us) log.cc:844] Serialized 173 byte log entry
0115 14:52:01.045862 (+1004697us) op_driver.cc:464] REPLICATION: finished
0115 14:52:01.046057 (+ 195us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0115 14:52:01.067989 (+ 21932us) log.cc:844] Serialized 5 byte log entry
0115 14:52:01.122815 (+ 54826us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.123052 (+ 237us) log.cc:844] Serialized 14 byte log entry
0115 14:52:01.123088 (+ 36us) alter_schema_op.cc:73] Released schema lock
0115 14:52:01.123092 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":142,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":24741,"prepare.run_wall_time_us":243049,"raft.queue_time_us":4912,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":8,"replication_time_us":1004830,"thread_start_us":242257,"threads_started":1,"wal-append.queue_time_us":250712}]]}
I20250115 14:52:01.192256 1167 ts_tablet_manager.cc:1905] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:01.192368 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 116747us
I20250115 14:52:01.193956 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 0291acf45d75421b8b1c6311560e7f0c (table default.Table_bbe00fc656b44633ae2fd159ae77ffc0 [id=e6f49fcfb7d74c71904b4e6b1bdfee1d]) successfully deleted
I20250115 14:52:01.194418 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c81f47cd6e4a308cd62124db38d42f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.195324 1887 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:01.195982 1162 tablet_bootstrap.cc:654] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.196022 1884 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 14:52:01.196889 1049 raft_consensus.cc:2463] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:01.198465 893 tablet_service.cc:1467] Processing CreateTablet for tablet f400880835ab4efdb2af0be5553d20dc (DEFAULT_TABLE table=default.Table_38b74aa452d14ca5b22b7259f85af561 [id=ce47660a249c41c68f6b6b2c2c9cd485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.198741 1012 tablet_service.cc:1467] Processing CreateTablet for tablet f400880835ab4efdb2af0be5553d20dc (DEFAULT_TABLE table=default.Table_38b74aa452d14ca5b22b7259f85af561 [id=ce47660a249c41c68f6b6b2c2c9cd485]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.199009 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f3f6de654d4765b1d7a381f93b0c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.199481 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5703fc5995184298bf72f9e43efb41f5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8e6cc08065494d9e85eeff5044b44ba6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.199772 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f60e68b411364ada9b89879012d862ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.200388 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ce16fea4f9469c9fc06fa24f6554be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.200753 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57aecd22f97c41618f90d14c2f53a7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.200776 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cae4c9228c640e495507e8f7ad9f544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.201565 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6cb8dfa7c42480bbd139d5acf059a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.202157 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 342bf8d4dc4244f982b1b63aaae0fb45 (DEFAULT_TABLE table=default.Table_1ee3b83d4fee4651b3979b44d4903db6 [id=1822bed451a74f2dbd54da029abd4384]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.202368 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0f3f6de654d4765b1d7a381f93b0c43. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.203017 1168 ts_tablet_manager.cc:1397] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.588s user 0.002s sys 0.022s
I20250115 14:52:01.204953 885 tablet_service.cc:1467] Processing CreateTablet for tablet 342bf8d4dc4244f982b1b63aaae0fb45 (DEFAULT_TABLE table=default.Table_1ee3b83d4fee4651b3979b44d4903db6 [id=1822bed451a74f2dbd54da029abd4384]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:01.205513 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4655600323f944969cfc4eaf5bbbb684 consensus state has no leader
I20250115 14:52:01.205962 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dd66b1c3bf36403bb0d9300ca8f318a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.206497 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd66b1c3bf36403bb0d9300ca8f318a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.207587 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0fd94e4a7fef44f5be462e7a77e8a123
I20250115 14:52:01.208182 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_13d7e84841db4e7897fa09136b16e197" } new_table_name: "default.Table_9edd581e8fc4457697465360c4504ff1" modify_external_catalogs: true
I20250115 14:52:01.208536 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f60e68b411364ada9b89879012d862ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.209815 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_903db046a0034d12a7be31cac011cab0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.210251 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_903db046a0034d12a7be31cac011cab0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.210471 847 leader_election.cc:304] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.210568 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 342bf8d4dc4244f982b1b63aaae0fb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.212095 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f400880835ab4efdb2af0be5553d20dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.212354 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 009ddfe9dfe14ba2b067f907a9d13538 consensus state has no leader
I20250115 14:52:01.212725 1808 fs_manager.cc:730] Time spent opening block manager: real 0.091s user 0.008s sys 0.010s
I20250115 14:52:01.212699 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5a8b86b253c04399a44480d59eeddea5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.213129 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a8b86b253c04399a44480d59eeddea5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.212911 1808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data,/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3"
format_stamp: "Formatted at 2025-01-15 14:51:48 on dist-test-slave-pxr4"
I20250115 14:52:01.214108 1808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 14:52:01.216784 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 6308f68e41554470b7d1321c60d4eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.216998 892 tablet_service.cc:1514] Processing DeleteTablet for tablet b5177265ba974ad6a381adb1e8c49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.217458 882 tablet_service.cc:1467] Processing CreateTablet for tablet 57aecd22f97c41618f90d14c2f53a7c9 (DEFAULT_TABLE table=default.Table_9118acb2bd5e43e9b5d4b0d2379fadfb [id=492ed43384bd463dbfa14c65b58044e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.218317 892 tablet_service.cc:1467] Processing CreateTablet for tablet 11fddbd9ba14443a91865387e1d1c238 (DEFAULT_TABLE table=default.Table_3f8db86f4b5b4f1b88804f296cc86c3f [id=311b894955f4455aa727f31c46879bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.219305 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd94e4a7fef44f5be462e7a77e8a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.219961 1887 tablet.cc:1721] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:01.222934 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f400880835ab4efdb2af0be5553d20dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.226744 1844 raft_consensus.cc:2798] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:01.226886 847 leader_election.cc:304] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:01.228179 1843 raft_consensus.cc:2798] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:01.228936 1168 raft_consensus.cc:357] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.230710 1844 raft_consensus.cc:695] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
W20250115 14:52:01.230772 848 consensus_peers.cc:487] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b90205ef15f470a99c013e7f2cf99c6. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:01.231020 1844 consensus_queue.cc:237] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.231657 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 6308f68e41554470b7d1321c60d4eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.232169 1843 raft_consensus.cc:491] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:01.232282 1843 raft_consensus.cc:3054] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:01.232988 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cae4c9228c640e495507e8f7ad9f544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.233140 1168 raft_consensus.cc:383] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.233309 1168 raft_consensus.cc:738] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.234346 1168 consensus_queue.cc:260] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.234902 1168 ts_tablet_manager.cc:1428] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250115 14:52:01.235893 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57aecd22f97c41618f90d14c2f53a7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.237478 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02ce16fea4f9469c9fc06fa24f6554be. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.238673 984 consensus_peers.cc:487] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5a5a92804cb462181e68e1cec02102b. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:01.239674 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet b5177265ba974ad6a381adb1e8c49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.239915 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 11fddbd9ba14443a91865387e1d1c238 (DEFAULT_TABLE table=default.Table_3f8db86f4b5b4f1b88804f296cc86c3f [id=311b894955f4455aa727f31c46879bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.242981 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6cb8dfa7c42480bbd139d5acf059a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.244352 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fddbd9ba14443a91865387e1d1c238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.244443 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 342bf8d4dc4244f982b1b63aaae0fb45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.244993 1019 tablet_service.cc:1467] Processing CreateTablet for tablet e6d67ddf01e34541bb795755e03c8249 (DEFAULT_TABLE table=default.Table_d495ddf9033f421e9d6cc7c06d08c8f2 [id=cf83ec5e8f3c4a0d92bdd962fbba9a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.245344 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6d67ddf01e34541bb795755e03c8249. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.246520 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd94e4a7fef44f5be462e7a77e8a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.240340 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 7718729182a340fe9b2dfea5a8321c1f (DEFAULT_TABLE table=default.Table_8234c8fb043a45b495995ce2bcd14cc7 [id=199476de8a584f9e9f6bcf826e83146b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:01.233104 848 leader_election.cc:343] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.249984 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7718729182a340fe9b2dfea5a8321c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.261143 875 tablet_service.cc:1467] Processing CreateTablet for tablet 5882252383ba46aba0ec053fab1fd383 (DEFAULT_TABLE table=default.Table_095514432af8489eab6e04a8b9bf1f53 [id=1532f36247a340cbba3c9b792d500c04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.262099 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fddbd9ba14443a91865387e1d1c238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.265106 878 tablet_service.cc:1467] Processing CreateTablet for tablet 7718729182a340fe9b2dfea5a8321c1f (DEFAULT_TABLE table=default.Table_8234c8fb043a45b495995ce2bcd14cc7 [id=199476de8a584f9e9f6bcf826e83146b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.271294 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5882252383ba46aba0ec053fab1fd383. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.271896 1168 tablet_bootstrap.cc:492] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:01.275831 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd94e4a7fef44f5be462e7a77e8a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.274307 1843 raft_consensus.cc:513] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.277069 1843 leader_election.cc:290] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:01.281023 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7718729182a340fe9b2dfea5a8321c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.282426 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c81f47cd6e4a308cd62124db38d42f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:01.282624 1049 raft_consensus.cc:3054] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:01.285154 1049 raft_consensus.cc:2463] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:01.287827 1876 raft_consensus.cc:491] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.287940 1876 raft_consensus.cc:513] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.288599 1876 leader_election.cc:290] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.290769 1168 tablet_bootstrap.cc:654] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.292748 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276abefcab6a422aa6467f00864c01f4" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.292775 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c81f47cd6e4a308cd62124db38d42f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:01.293373 984 leader_election.cc:343] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.296166 847 leader_election.cc:304] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:01.300446 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276abefcab6a422aa6467f00864c01f4" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:01.301333 983 leader_election.cc:343] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.301462 983 leader_election.cc:304] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.301862 1876 raft_consensus.cc:2743] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.302255 874 tablet_service.cc:1467] Processing CreateTablet for tablet e6d67ddf01e34541bb795755e03c8249 (DEFAULT_TABLE table=default.Table_d495ddf9033f421e9d6cc7c06d08c8f2 [id=cf83ec5e8f3c4a0d92bdd962fbba9a67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.302670 1843 raft_consensus.cc:2798] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:01.302832 1843 raft_consensus.cc:695] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:01.303095 1843 consensus_queue.cc:237] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.303799 1882 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:01.306496 1865 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250115 14:52:01.311545 848 leader_election.cc:343] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.315408 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6d67ddf01e34541bb795755e03c8249. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.318650 1865 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:43552 (request call id 509) took 1034 ms. Trace:
I20250115 14:52:01.318805 1865 rpcz_store.cc:276] 0115 14:52:00.283725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.286081 (+ 2356us) service_pool.cc:225] Handling call
0115 14:52:01.318631 (+1032550us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0115 14:52:00.286497 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0115 14:52:00.286567 (+ 70us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0115 14:52:01.124599 (+838032us) alter_schema_op.cc:66] Acquired schema lock
0115 14:52:01.124610 (+ 11us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0115 14:52:01.124671 (+ 61us) alter_schema_op.cc:128] START. Timestamp: 1736952721124627
0115 14:52:01.124674 (+ 3us) op_driver.cc:348] REPLICATION: starting
0115 14:52:01.124835 (+ 161us) log.cc:844] Serialized 173 byte log entry
0115 14:52:01.302698 (+177863us) op_driver.cc:464] REPLICATION: finished
0115 14:52:01.306479 (+ 3781us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0115 14:52:01.306829 (+ 350us) log.cc:844] Serialized 5 byte log entry
0115 14:52:01.314001 (+ 7172us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.315767 (+ 1766us) log.cc:844] Serialized 14 byte log entry
0115 14:52:01.318579 (+ 2812us) alter_schema_op.cc:73] Released schema lock
0115 14:52:01.318588 (+ 9us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":3726,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":56908,"prepare.run_wall_time_us":838411,"replication_time_us":177994}]]}
I20250115 14:52:01.334398 1751 ts_tablet_manager.cc:1918] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:01.335937 1751 log.cc:1198] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9f327ffa731c4c949b24a34dc54ae260
I20250115 14:52:01.336558 1751 ts_tablet_manager.cc:1939] T 9f327ffa731c4c949b24a34dc54ae260 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:01.341799 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 472) took 1493 ms. Trace:
I20250115 14:52:01.342537 1168 tablet_bootstrap.cc:492] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:01.342723 1168 ts_tablet_manager.cc:1397] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.003s
I20250115 14:52:01.345598 871 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56242 (request call id 115) took 1147 ms. Trace:
I20250115 14:52:01.346027 1751 rpcz_store.cc:276] 0115 14:51:59.848465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.848515 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:00.999656 (+1151141us) ts_tablet_manager.cc:1145] Deleting tablet 9f327ffa731c4c949b24a34dc54ae260
0115 14:52:00.999667 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.046642 (+ 46975us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x616000220354. stack: 00007f9adcf568a6 00007f9adcf56660 00007f9adbf3b48d 000000000096d544 000000000096d50d 000000000096ceda 00007f9ae87f7ed0 00007f9ae9aec463 00007f9aec590f67 00007f9aec5c3634 00007f9aec5a4740 00007f9aec5a44cd 0000000000a71ecc 00007f9adcf9d434 00007f9adcfa0a5b 00007f9adcfa08d8
0115 14:52:01.309682 (+263040us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.334294 (+ 24612us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.341778 (+ 7484us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68313344,"tablet-delete.queue_time_us":1148990}
I20250115 14:52:01.345425 1168 raft_consensus.cc:357] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.347744 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 474) took 1483 ms. Trace:
I20250115 14:52:01.345750 871 rpcz_store.cc:276] 0115 14:52:00.176038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.181561 (+ 5523us) service_pool.cc:225] Handling call
0115 14:52:01.206056 (+1024495us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x616000162754. stack: 00007f04431e08a6 00007f04431e0660 00007f04421c548d 000000000096d544 000000000096d50d 000000000096ceda 00007f044ea51fd0 00007f045278ebef 00007f044d0e98e6 00007f044d0e9615 00007f0449f4630e 00007f0449f451ee 00007f0449f4b425 00007f0449f4bd4f 00007f0449f4bb78 0000000000a71ecc
0115 14:52:01.228677 (+ 22621us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x6160000b1b30. stack: 00007f04431e08a6 00007f04431e0660 00007f04421c548d 000000000096d544 000000000096d50d 000000000096ceda 00007f044fd80068 00007f045278e9f8 00007f044d0e98e6 00007f044d0e9615 00007f0449f4630e 00007f0449f451ee 00007f0449f4b425 00007f0449f4bd4f 00007f0449f4bb78 0000000000a71ecc
0115 14:52:01.242862 (+ 14185us) spinlock_profiling.cc:243] Waited 4.07 ms on lock 0x6160000a2230. stack: 00007f04431e08a6 00007f04431e0660 00007f04421c548d 000000000096d544 000000000096d50d 000000000096ceda 00007f044fd80068 00007f045278e9f8 00007f044d0e98e6 00007f044d0e9615 00007f0449f4630e 00007f0449f451ee 00007f0449f4b425 00007f0449f4bd4f 00007f0449f4bb78 0000000000a71ecc
0115 14:52:01.323075 (+ 80213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":319531136}
I20250115 14:52:01.348136 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9f327ffa731c4c949b24a34dc54ae260 (table default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]) successfully deleted
I20250115 14:52:01.349231 1751 rpcz_store.cc:276] 0115 14:51:59.864283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.864660 (+ 377us) service_pool.cc:225] Handling call
0115 14:52:01.347593 (+1482933us) ts_tablet_manager.cc:1145] Deleting tablet 248cadcdef8b4e99923cdbca39690685
0115 14:52:01.347603 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.347717 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482690}
I20250115 14:52:01.349838 1751 tablet_replica.cc:331] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:01.350193 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 248cadcdef8b4e99923cdbca39690685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 248cadcdef8b4e99923cdbca39690685 already in progress: creating tablet
I20250115 14:52:01.354334 1168 raft_consensus.cc:383] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.356562 1751 raft_consensus.cc:2238] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.357023 1751 raft_consensus.cc:2267] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.362712 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8edddd03b4f34de4bb4801a6dcca183b (DEFAULT_TABLE table=default.Table_116f78c6117c4cf4b8db61d75ae62013 [id=5c127085c24a42688f4d0f2808d6ffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.363145 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8edddd03b4f34de4bb4801a6dcca183b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.367872 1168 raft_consensus.cc:738] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.368124 1168 consensus_queue.cc:260] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.368750 1168 ts_tablet_manager.cc:1428] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250115 14:52:01.369036 1168 tablet_bootstrap.cc:492] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:01.346961 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 7d612b424bea408cb6691c7c200cf3a6 (DEFAULT_TABLE table=default.Table_ec781cbe91bf4224b2b7c28a98ccac67 [id=7573bfd9285b4b51babf235b0e4bb310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.369469 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d612b424bea408cb6691c7c200cf3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.371368 889 tablet_service.cc:1467] Processing CreateTablet for tablet d18a88c8f3ce4a24b7e1e13966760364 (DEFAULT_TABLE table=default.Table_3d9f1f67737c458aa1d0004f77e93808 [id=7ed97c31045b49d7b417f7e773dd2a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.340509 1019 tablet_service.cc:1467] Processing CreateTablet for tablet d18a88c8f3ce4a24b7e1e13966760364 (DEFAULT_TABLE table=default.Table_3d9f1f67737c458aa1d0004f77e93808 [id=7ed97c31045b49d7b417f7e773dd2a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.372829 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d18a88c8f3ce4a24b7e1e13966760364. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.342211 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 5882252383ba46aba0ec053fab1fd383 (DEFAULT_TABLE table=default.Table_095514432af8489eab6e04a8b9bf1f53 [id=1532f36247a340cbba3c9b792d500c04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.376463 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5882252383ba46aba0ec053fab1fd383. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.345753 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b8589265cc054407b68055bf5665057e (DEFAULT_TABLE table=default.Table_3657238a564a4fec99af530ff70671af [id=ce06e03a310c40f2a3b2883f2bf45b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.379698 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8589265cc054407b68055bf5665057e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.340678 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 8edddd03b4f34de4bb4801a6dcca183b (DEFAULT_TABLE table=default.Table_116f78c6117c4cf4b8db61d75ae62013 [id=5c127085c24a42688f4d0f2808d6ffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.380606 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8edddd03b4f34de4bb4801a6dcca183b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.380856 876 tablet_service.cc:1467] Processing CreateTablet for tablet 4655600323f944969cfc4eaf5bbbb684 (DEFAULT_TABLE table=default.Table_76ced15792e04bd0890267d54f928393 [id=d0eba937a20844a484d0bdc69ef71524]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.390533 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d18a88c8f3ce4a24b7e1e13966760364. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.346547 1015 tablet_service.cc:1467] Processing CreateTablet for tablet c14836b97c3247b39ff9072c12e642e0 (DEFAULT_TABLE table=default.Table_7c155cee1f6e49a99d8ab38956a737e2 [id=e5490ddb05a04c27ab00cfd7ad2d3747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.391299 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c14836b97c3247b39ff9072c12e642e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.393961 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4655600323f944969cfc4eaf5bbbb684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.395344 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 4655600323f944969cfc4eaf5bbbb684 (DEFAULT_TABLE table=default.Table_76ced15792e04bd0890267d54f928393 [id=d0eba937a20844a484d0bdc69ef71524]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.395756 1168 tablet_bootstrap.cc:654] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.395773 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4655600323f944969cfc4eaf5bbbb684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.397504 1751 ts_tablet_manager.cc:1905] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.398484 883 tablet_service.cc:1467] Processing CreateTablet for tablet c14836b97c3247b39ff9072c12e642e0 (DEFAULT_TABLE table=default.Table_7c155cee1f6e49a99d8ab38956a737e2 [id=e5490ddb05a04c27ab00cfd7ad2d3747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.400164 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c14836b97c3247b39ff9072c12e642e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.413393 878 tablet_service.cc:1467] Processing CreateTablet for tablet 7d612b424bea408cb6691c7c200cf3a6 (DEFAULT_TABLE table=default.Table_ec781cbe91bf4224b2b7c28a98ccac67 [id=7573bfd9285b4b51babf235b0e4bb310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.413890 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d612b424bea408cb6691c7c200cf3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.425727 1162 tablet_bootstrap.cc:492] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.425968 1162 ts_tablet_manager.cc:1397] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.257s user 0.007s sys 0.005s
I20250115 14:52:01.426765 1162 raft_consensus.cc:357] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.427048 1162 raft_consensus.cc:383] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.427150 1162 raft_consensus.cc:738] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.427405 1162 consensus_queue.cc:260] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.428077 1162 ts_tablet_manager.cc:1428] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250115 14:52:01.432417 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:40227, user_credentials={real_user=slave}} blocked reactor thread for 104002us
I20250115 14:52:01.433745 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 104282a09c024bbaafeb5f28f91dadcb
I20250115 14:52:01.435400 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_2e2abefaccf54e469bacf8126edad0c4 [id=855145809143431d81790dd36147a77b]). Expected version 2 got 1
I20250115 14:52:01.435544 1751 ts_tablet_manager.cc:1918] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.435607 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 1839edd0d49b467d8bb31b3001ff89eb (table default.Table_253ac9c181914d0f891baf1f8722dbd0 [id=a724f6e4ad16485b8ee2445bb072be3e]). Expected version 1 got 0
I20250115 14:52:01.435642 1751 log.cc:1198] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b92231509c754e8dbfc564813a76a123
I20250115 14:52:01.435923 1751 ts_tablet_manager.cc:1939] T b92231509c754e8dbfc564813a76a123 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:01.436650 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 475) took 1511 ms. Trace:
I20250115 14:52:01.437223 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5dd9ae0549364620b988ae28f376f552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.437883 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dd9ae0549364620b988ae28f376f552 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.437788 1162 tablet_bootstrap.cc:492] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.446988 1162 tablet_bootstrap.cc:654] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.440451 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 104282a09c024bbaafeb5f28f91dadcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.448041 1750 raft_consensus.cc:491] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.448006 1751 rpcz_store.cc:276] 0115 14:51:59.925231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.925577 (+ 346us) service_pool.cc:225] Handling call
0115 14:52:01.349790 (+1424213us) ts_tablet_manager.cc:1145] Deleting tablet b92231509c754e8dbfc564813a76a123
0115 14:52:01.349799 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.403319 (+ 53520us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.435441 (+ 32122us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.436631 (+ 1190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1423992}
I20250115 14:52:01.448199 1750 raft_consensus.cc:513] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.448436 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b92231509c754e8dbfc564813a76a123 (table default.Table_02593da1cd7245f38f85424d1a4baad9 [id=1b38e621a58442c88f40bb9296563c36]) successfully deleted
I20250115 14:52:01.448928 1751 tablet_replica.cc:331] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:01.449096 1751 raft_consensus.cc:2238] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.449242 1751 raft_consensus.cc:2267] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.449710 1751 ts_tablet_manager.cc:1905] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.450453 1164 tablet_bootstrap.cc:492] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:01.454031 1750 leader_election.cc:290] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.457171 741 tablet_service.cc:1467] Processing CreateTablet for tablet d18a88c8f3ce4a24b7e1e13966760364 (DEFAULT_TABLE table=default.Table_3d9f1f67737c458aa1d0004f77e93808 [id=7ed97c31045b49d7b417f7e773dd2a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.457580 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d18a88c8f3ce4a24b7e1e13966760364. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.458357 749 tablet_service.cc:1467] Processing CreateTablet for tablet 8edddd03b4f34de4bb4801a6dcca183b (DEFAULT_TABLE table=default.Table_116f78c6117c4cf4b8db61d75ae62013 [id=5c127085c24a42688f4d0f2808d6ffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.458945 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8edddd03b4f34de4bb4801a6dcca183b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.460057 739 tablet_service.cc:1467] Processing CreateTablet for tablet c14836b97c3247b39ff9072c12e642e0 (DEFAULT_TABLE table=default.Table_7c155cee1f6e49a99d8ab38956a737e2 [id=e5490ddb05a04c27ab00cfd7ad2d3747]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.460415 745 tablet_service.cc:1467] Processing CreateTablet for tablet b8589265cc054407b68055bf5665057e (DEFAULT_TABLE table=default.Table_3657238a564a4fec99af530ff70671af [id=ce06e03a310c40f2a3b2883f2bf45b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.460734 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8589265cc054407b68055bf5665057e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:01.461294 710 leader_election.cc:343] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 5703fc5995184298bf72f9e43efb41f5
I20250115 14:52:01.461361 743 tablet_service.cc:1467] Processing CreateTablet for tablet 7d612b424bea408cb6691c7c200cf3a6 (DEFAULT_TABLE table=default.Table_ec781cbe91bf4224b2b7c28a98ccac67 [id=7573bfd9285b4b51babf235b0e4bb310]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.461678 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d612b424bea408cb6691c7c200cf3a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.462556 752 tablet_service.cc:1467] Processing CreateTablet for tablet 4655600323f944969cfc4eaf5bbbb684 (DEFAULT_TABLE table=default.Table_76ced15792e04bd0890267d54f928393 [id=d0eba937a20844a484d0bdc69ef71524]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.462857 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4655600323f944969cfc4eaf5bbbb684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.463164 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 104282a09c024bbaafeb5f28f91dadcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.469817 1164 ts_tablet_manager.cc:1397] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.374s user 0.004s sys 0.003s
I20250115 14:52:01.470577 1164 raft_consensus.cc:357] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.470816 1164 raft_consensus.cc:383] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.470885 1164 raft_consensus.cc:738] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.474195 1751 ts_tablet_manager.cc:1918] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.474346 1167 ts_tablet_manager.cc:1918] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.474565 1167 log.cc:1198] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/79aad920dd9744d9b87a8234c92232d8
I20250115 14:52:01.474967 1167 ts_tablet_manager.cc:1939] T 79aad920dd9744d9b87a8234c92232d8 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:01.475313 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 436) took 2018 ms. Trace:
I20250115 14:52:01.475467 1167 rpcz_store.cc:276] 0115 14:51:59.457075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.460859 (+ 3784us) service_pool.cc:225] Handling call
0115 14:52:01.170306 (+1709447us) ts_tablet_manager.cc:1145] Deleting tablet 79aad920dd9744d9b87a8234c92232d8
0115 14:52:01.170317 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.347006 (+176689us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.454050 (+107044us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.475282 (+ 21232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1709251}
I20250115 14:52:01.475872 1167 tablet_replica.cc:331] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:01.476130 1167 raft_consensus.cc:2238] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.476739 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c14836b97c3247b39ff9072c12e642e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.480638 1162 tablet_bootstrap.cc:492] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.480890 1162 ts_tablet_manager.cc:1397] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.006s
I20250115 14:52:01.481552 1162 raft_consensus.cc:357] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.481783 1162 raft_consensus.cc:383] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.481863 1162 raft_consensus.cc:738] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.482092 1162 consensus_queue.cc:260] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.482687 1162 ts_tablet_manager.cc:1428] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:01.482982 1162 tablet_bootstrap.cc:492] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.484545 1162 tablet_bootstrap.cc:654] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.485368 1751 log.cc:1198] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/bd4102b1f83e414d93a6a68315e36d16
I20250115 14:52:01.485706 1751 ts_tablet_manager.cc:1939] T bd4102b1f83e414d93a6a68315e36d16 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:01.486452 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 476) took 1561 ms. Trace:
I20250115 14:52:01.486573 1751 rpcz_store.cc:276] 0115 14:51:59.925344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.925831 (+ 487us) service_pool.cc:225] Handling call
0115 14:52:01.448884 (+1523053us) ts_tablet_manager.cc:1145] Deleting tablet bd4102b1f83e414d93a6a68315e36d16
0115 14:52:01.448894 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.455876 (+ 6982us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.474115 (+ 18239us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.486434 (+ 12319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1522878}
I20250115 14:52:01.487085 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet bd4102b1f83e414d93a6a68315e36d16 (table default.Table_d0d86551c8474fb98ad74cb89ff84e28 [id=becd439cb45d48c3974b9ae72768542a]) successfully deleted
I20250115 14:52:01.487519 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 477) took 1562 ms. Trace:
I20250115 14:52:01.487627 1751 rpcz_store.cc:276] 0115 14:51:59.925435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.926038 (+ 603us) service_pool.cc:225] Handling call
0115 14:52:01.487405 (+1561367us) ts_tablet_manager.cc:1145] Deleting tablet 4294c49da1e044be993100b3cf5e2cea
0115 14:52:01.487413 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.487503 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561207}
W20250115 14:52:01.488102 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 4294c49da1e044be993100b3cf5e2cea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4294c49da1e044be993100b3cf5e2cea already in progress: creating tablet
I20250115 14:52:01.488482 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 478) took 1562 ms. Trace:
I20250115 14:52:01.488575 1751 rpcz_store.cc:276] 0115 14:51:59.925529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.926218 (+ 689us) service_pool.cc:225] Handling call
0115 14:52:01.488395 (+1562177us) ts_tablet_manager.cc:1145] Deleting tablet 6e616d78933e4f84b9ec2acc51450b18
0115 14:52:01.488402 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.488469 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562028}
W20250115 14:52:01.488986 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6e616d78933e4f84b9ec2acc51450b18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e616d78933e4f84b9ec2acc51450b18 already in progress: creating tablet
I20250115 14:52:01.489337 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 481) took 1481 ms. Trace:
I20250115 14:52:01.489440 1751 rpcz_store.cc:276] 0115 14:52:00.008223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.008270 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:01.489243 (+1480973us) ts_tablet_manager.cc:1145] Deleting tablet 43605c995df4400e91139199c9514bfd
0115 14:52:01.489252 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.489322 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1480661}
W20250115 14:52:01.489871 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 43605c995df4400e91139199c9514bfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43605c995df4400e91139199c9514bfd already in progress: creating tablet
I20250115 14:52:01.490159 1751 tablet_replica.cc:331] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:01.490350 1751 raft_consensus.cc:2238] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.490518 1751 raft_consensus.cc:2267] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.490759 1166 ts_tablet_manager.cc:1918] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.490907 1166 log.cc:1198] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/a73bd0141ccc4ec4b052b18b9fb62bb7
I20250115 14:52:01.491220 1728 raft_consensus.cc:491] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.491264 1166 ts_tablet_manager.cc:1939] T a73bd0141ccc4ec4b052b18b9fb62bb7 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:01.491300 1751 ts_tablet_manager.cc:1905] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.491334 1728 raft_consensus.cc:513] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.491899 1728 leader_election.cc:290] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.492163 1166 tablet_replica.cc:331] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:01.492658 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c385edf4414157845b971aaf366831" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:01.493147 984 leader_election.cc:343] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.493165 1164 consensus_queue.cc:260] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.493741 1164 ts_tablet_manager.cc:1428] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.024s user 0.000s sys 0.001s
I20250115 14:52:01.494690 1164 tablet_bootstrap.cc:492] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:01.498064 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 104282a09c024bbaafeb5f28f91dadcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.498704 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 79aad920dd9744d9b87a8234c92232d8 (table default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]) successfully deleted
W20250115 14:52:01.502750 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:45653, user_credentials={real_user=slave}} blocked reactor thread for 57458.7us
I20250115 14:52:01.503772 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet a73bd0141ccc4ec4b052b18b9fb62bb7 (table default.Table_44a23f5fdfb446a59f551a2a7a4ca2ce [id=9a5faae475a948528f47d259a547189b]) successfully deleted
W20250115 14:52:01.504398 441 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b -> Peer 7d97cf5f1fbb4bc6883711020e2fa3f3 (127.0.104.61:45653): Couldn't send request to peer 7d97cf5f1fbb4bc6883711020e2fa3f3. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.61:45653: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250115 14:52:01.504149 1162 tablet_bootstrap.cc:492] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.504688 1162 ts_tablet_manager.cc:1397] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.004s
I20250115 14:52:01.505333 1162 raft_consensus.cc:357] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.505493 1162 raft_consensus.cc:383] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.505537 1162 raft_consensus.cc:738] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.505733 1162 consensus_queue.cc:260] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.506269 1162 ts_tablet_manager.cc:1428] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:01.506970 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb] alter complete (version 1)
I20250115 14:52:01.507939 1162 tablet_bootstrap.cc:492] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.509469 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3f541e916d264738842cd965c7a2a5ec
I20250115 14:52:01.509675 1162 tablet_bootstrap.cc:654] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.510160 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ab10a36d2dd14df0b5b7dde01478a485"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:01.510531 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f327ffa731c4c949b24a34dc54ae260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:01.510751 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab10a36d2dd14df0b5b7dde01478a485 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.511462 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 276abefcab6a422aa6467f00864c01f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.511615 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f541e916d264738842cd965c7a2a5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.517515 1751 ts_tablet_manager.cc:1918] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.517604 1751 log.cc:1198] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/0de6f6f014074c3d8775cb5e4121ccb0
I20250115 14:52:01.517884 1751 ts_tablet_manager.cc:1939] T 0de6f6f014074c3d8775cb5e4121ccb0 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:01.518515 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 483) took 1458 ms. Trace:
I20250115 14:52:01.518617 1751 rpcz_store.cc:276] 0115 14:52:00.060111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.060423 (+ 312us) service_pool.cc:225] Handling call
0115 14:52:01.490119 (+1429696us) ts_tablet_manager.cc:1145] Deleting tablet 0de6f6f014074c3d8775cb5e4121ccb0
0115 14:52:01.490128 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.493965 (+ 3837us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.517431 (+ 23466us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.518500 (+ 1069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429478}
I20250115 14:52:01.519030 1751 tablet_replica.cc:331] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:01.519019 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 0de6f6f014074c3d8775cb5e4121ccb0 (table default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]) successfully deleted
I20250115 14:52:01.521448 1751 raft_consensus.cc:2238] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.521617 1751 raft_consensus.cc:2267] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.522094 1751 ts_tablet_manager.cc:1905] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.526773 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:52:01.527776 1162 tablet_bootstrap.cc:492] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.527943 1162 ts_tablet_manager.cc:1397] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
I20250115 14:52:01.528683 1162 raft_consensus.cc:357] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.528936 1162 raft_consensus.cc:383] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.529028 1162 raft_consensus.cc:738] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.529258 1162 consensus_queue.cc:260] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.529477 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 496dec75500347699625a755dceed2bb
I20250115 14:52:01.529898 1162 ts_tablet_manager.cc:1428] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:01.530478 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 496dec75500347699625a755dceed2bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:01.530504 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b76d1bbf90c6416ca60805c9d8053bd4 consensus state has no leader
I20250115 14:52:01.530949 1162 tablet_bootstrap.cc:492] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.530988 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 276abefcab6a422aa6467f00864c01f4 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:01.531625 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a2fe51b51994491ab38a644bfc55e3ac
I20250115 14:52:01.532357 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fe51b51994491ab38a644bfc55e3ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.532433 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 496dec75500347699625a755dceed2bb from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:01.532745 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ab69195bec7c448888d88776d4576735"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.533248 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab69195bec7c448888d88776d4576735 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.533506 1162 tablet_bootstrap.cc:654] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.534298 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_57cca7a5af0b434b92b6e5c6eb5413f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.534790 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57cca7a5af0b434b92b6e5c6eb5413f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.536980 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7488025a06ba42b5814d9788890cf2a0" } new_table_name: "default.Table_efb7e019ddea47a8954081c1715654e3" modify_external_catalogs: true
I20250115 14:52:01.538986 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_53f051fa3f6b4051882de44c9baff2aa" } new_table_name: "default.Table_d548be616a2c4f01ac52eadd5c009eaf" modify_external_catalogs: true
I20250115 14:52:01.539345 1751 ts_tablet_manager.cc:1918] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.539450 1751 log.cc:1198] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/629873c00550408798b00489ece4ea5f
I20250115 14:52:01.539716 1751 ts_tablet_manager.cc:1939] T 629873c00550408798b00489ece4ea5f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:01.540470 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 484) took 1472 ms. Trace:
I20250115 14:52:01.540555 1751 rpcz_store.cc:276] 0115 14:52:00.068283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.068386 (+ 103us) service_pool.cc:225] Handling call
0115 14:52:01.518991 (+1450605us) ts_tablet_manager.cc:1145] Deleting tablet 629873c00550408798b00489ece4ea5f
0115 14:52:01.519000 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.528962 (+ 9962us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.539243 (+ 10281us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.540450 (+ 1207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1450395}
I20250115 14:52:01.540856 1751 tablet_replica.cc:331] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:01.541033 1751 raft_consensus.cc:2238] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.541160 1751 raft_consensus.cc:2267] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.541625 1751 ts_tablet_manager.cc:1905] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.543639 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_2e2abefaccf54e469bacf8126edad0c4 [id=855145809143431d81790dd36147a77b] alter complete (version 2)
I20250115 14:52:01.543776 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 629873c00550408798b00489ece4ea5f (table default.Table_992f266d1ab24ee68d346afbfdec91eb [id=dfc41804229046b29a6d1230b41b9fa7]) successfully deleted
I20250115 14:52:01.544251 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 010e6535b6bc44f391e6f2b6f1686532 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:01.545432 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dd281cfe0677416e8a04ee13689d050f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:01.545774 1865 tablet.cc:1721] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250115 14:52:01.545862 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd281cfe0677416e8a04ee13689d050f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.546213 1808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 14:52:01.547281 1808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 14:52:01.547547 1808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 14:52:01.551235 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c3b50255eb3e4642abaa02355b819e25" } new_table_name: "default.Table_87cf8b44f8fb4fbda80f72656bfaa074" modify_external_catalogs: true
I20250115 14:52:01.552857 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 87026b6faf75437eac21dedab1f41ed4 (DEFAULT_TABLE table=default.Table_e9bb748dd0854fe6ab5fb3dad43da484 [id=156506418e9d4322b5772bc57b409552]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.553185 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87026b6faf75437eac21dedab1f41ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.553299 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_02e5510e4edc458bab64cbb11e942275"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.553738 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02e5510e4edc458bab64cbb11e942275 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.554373 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5d8548b30284461086fdcd351454ad57" } modify_external_catalogs: true
I20250115 14:52:01.556056 1166 raft_consensus.cc:2238] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.556574 1166 raft_consensus.cc:2267] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.556944 875 tablet_service.cc:1467] Processing CreateTablet for tablet b8589265cc054407b68055bf5665057e (DEFAULT_TABLE table=default.Table_3657238a564a4fec99af530ff70671af [id=ce06e03a310c40f2a3b2883f2bf45b5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.557251 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3be094dcd3cb4f9d9259d354300c1600 (DEFAULT_TABLE table=default.Table_d050f7c481a549248abc7c8beb08de6e [id=387d14a5bd4143bfbcac7e58f2c37511]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.557535 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3be094dcd3cb4f9d9259d354300c1600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.559304 1751 ts_tablet_manager.cc:1918] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.559381 1751 log.cc:1198] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d97c2f443ab040f9bc1924f45379165e
I20250115 14:52:01.559638 1751 ts_tablet_manager.cc:1939] T d97c2f443ab040f9bc1924f45379165e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:01.559980 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c385edf4414157845b971aaf366831" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:01.562642 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 496dec75500347699625a755dceed2bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.562942 741 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fe51b51994491ab38a644bfc55e3ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.563921 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 496dec75500347699625a755dceed2bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.564076 1166 ts_tablet_manager.cc:1905] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.566593 1164 tablet_bootstrap.cc:654] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.560266 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 488) took 1300 ms. Trace:
W20250115 14:52:01.560528 983 leader_election.cc:343] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.567732 983 leader_election.cc:304] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.568259 749 tablet_service.cc:1467] Processing CreateTablet for tablet 87026b6faf75437eac21dedab1f41ed4 (DEFAULT_TABLE table=default.Table_e9bb748dd0854fe6ab5fb3dad43da484 [id=156506418e9d4322b5772bc57b409552]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.568634 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87026b6faf75437eac21dedab1f41ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.569165 741 tablet_service.cc:1467] Processing CreateTablet for tablet 3be094dcd3cb4f9d9259d354300c1600 (DEFAULT_TABLE table=default.Table_d050f7c481a549248abc7c8beb08de6e [id=387d14a5bd4143bfbcac7e58f2c37511]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.571439 1876 raft_consensus.cc:2743] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.577491 1168 tablet_bootstrap.cc:492] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:01.584592 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3be094dcd3cb4f9d9259d354300c1600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.585182 776 raft_consensus.cc:3054] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:01.588052 1167 raft_consensus.cc:2267] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.589980 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f541e916d264738842cd965c7a2a5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.591346 887 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fe51b51994491ab38a644bfc55e3ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.592792 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f541e916d264738842cd965c7a2a5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.592087 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5703fc5995184298bf72f9e43efb41f5" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:01.594040 1866 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:01.595233 872 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56242 (request call id 118) took 1184 ms. Trace:
I20250115 14:52:01.595604 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8589265cc054407b68055bf5665057e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.596464 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3be094dcd3cb4f9d9259d354300c1600 (DEFAULT_TABLE table=default.Table_d050f7c481a549248abc7c8beb08de6e [id=387d14a5bd4143bfbcac7e58f2c37511]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.597488 710 leader_election.cc:304] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:52:01.597712 710 raft_consensus.cc:2681] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:52:01.597977 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3be094dcd3cb4f9d9259d354300c1600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.598006 1883 tablet.cc:1721] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:01.567453 1751 rpcz_store.cc:276] 0115 14:52:00.259947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.260122 (+ 175us) service_pool.cc:225] Handling call
0115 14:52:01.540813 (+1280691us) ts_tablet_manager.cc:1145] Deleting tablet d97c2f443ab040f9bc1924f45379165e
0115 14:52:01.540823 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.556706 (+ 15883us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.559229 (+ 2523us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.560250 (+ 1021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1271183}
I20250115 14:52:01.601027 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d97c2f443ab040f9bc1924f45379165e (table default.Table_121cfa7da22441d581d821791e428785 [id=470df20dd8bd47f4acfa2d40f2e034bd]) successfully deleted
I20250115 14:52:01.596822 884 tablet_service.cc:1467] Processing CreateTablet for tablet 87026b6faf75437eac21dedab1f41ed4 (DEFAULT_TABLE table=default.Table_e9bb748dd0854fe6ab5fb3dad43da484 [id=156506418e9d4322b5772bc57b409552]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.607412 1168 ts_tablet_manager.cc:1397] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.209s user 0.006s sys 0.005s
I20250115 14:52:01.607847 872 rpcz_store.cc:276] 0115 14:52:00.388244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.388317 (+ 73us) service_pool.cc:225] Handling call
0115 14:52:01.067715 (+679398us) spinlock_profiling.cc:243] Waited 8.4 ms on lock 0x616000034730. stack: 00007f04431e08a6 00007f04431e0660 00007f04421c548d 000000000096d544 000000000096d50d 000000000096ceda 00007f044fd80068 00007f045278e9f8 00007f044d0e98e6 00007f044d0e9615 00007f0449f4630e 00007f0449f451ee 00007f0449f4b425 00007f0449f4bd4f 00007f0449f4bb78 0000000000a71ecc
0115 14:52:01.573091 (+505376us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19327232}
I20250115 14:52:01.622367 1166 ts_tablet_manager.cc:1918] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.622488 1166 log.cc:1198] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/5703fc5995184298bf72f9e43efb41f5
I20250115 14:52:01.622864 1166 ts_tablet_manager.cc:1939] T 5703fc5995184298bf72f9e43efb41f5 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:01.623093 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87026b6faf75437eac21dedab1f41ed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.624315 1167 ts_tablet_manager.cc:1905] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.625888 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 487) took 1501 ms. Trace:
I20250115 14:52:01.626427 1751 rpcz_store.cc:276] 0115 14:52:00.101067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.121295 (+ 20228us) service_pool.cc:225] Handling call
0115 14:52:01.601534 (+1480239us) ts_tablet_manager.cc:1145] Deleting tablet 8f8bae9867d745afb1dfdf47bcb881da
0115 14:52:01.601638 (+ 104us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.602276 (+ 638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1320797}
W20250115 14:52:01.627211 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 8f8bae9867d745afb1dfdf47bcb881da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f8bae9867d745afb1dfdf47bcb881da already in progress: creating tablet
I20250115 14:52:01.627612 1751 tablet_replica.cc:331] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:01.629372 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c3bef2a77d0640848112f55aa0ed662b" } new_table_name: "default.Table_f85e432eb37548eb9d03942617305020" modify_external_catalogs: true
W20250115 14:52:01.630674 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65a6f83730f74d13823bd15d93538df7 consensus state has no leader
I20250115 14:52:01.631055 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5d8548b30284461086fdcd351454ad57" } modify_external_catalogs: true
I20250115 14:52:01.631978 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fbde011216426cb41c51fe359dcb5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.632220 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cba5cc51d7414a9352caedcc362c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.632414 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c385edf4414157845b971aaf366831 with delete_type TABLET_DATA_DELETED (replaced by tablet 84844b5f50af408ab7dda73f7421cd9d) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.632601 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c81f47cd6e4a308cd62124db38d42f with delete_type TABLET_DATA_DELETED (replaced by tablet 580742eb5dac4dba9995809890e0e315) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.627888 1751 raft_consensus.cc:2238] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:01.634197 1751 pending_rounds.cc:62] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:52:01.634310 1751 pending_rounds.cc:69] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558345109864448 op_type: NO_OP noop_request { }
I20250115 14:52:01.634485 1751 raft_consensus.cc:2883] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:01.634626 1751 raft_consensus.cc:2267] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.635802 1168 raft_consensus.cc:357] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.636248 1882 tablet.cc:1721] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:01.636766 1168 raft_consensus.cc:383] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.636864 1168 raft_consensus.cc:738] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.644974 870 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:56242 (request call id 116) took 1304 ms. Trace:
I20250115 14:52:01.645105 1168 consensus_queue.cc:260] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.648168 870 rpcz_store.cc:276] 0115 14:52:00.337132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.341221 (+ 4089us) service_pool.cc:225] Handling call
0115 14:52:01.498901 (+1157680us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x616000174154. stack: 00007f04431e08a6 00007f04431e0660 00007f04421c548d 000000000096d544 000000000096d50d 000000000096ceda 00007f044ea51fd0 00007f045278ebef 00007f044d0e98e6 00007f044d0e9615 00007f0449f4630e 00007f0449f451ee 00007f0449f4b425 00007f0449f4bd4f 00007f0449f4bb78 0000000000a71ecc
0115 14:52:01.641612 (+142711us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":465188608}
I20250115 14:52:01.650187 1168 ts_tablet_manager.cc:1428] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20250115 14:52:01.650475 1168 tablet_bootstrap.cc:492] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:01.653525 1751 ts_tablet_manager.cc:1905] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.670514 451 master_service.cc:946] ReplaceTablet: received request to replace tablet fe696560ba6b4a76ad013d9a3486e398 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:01.673386 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bb07dc16172246278fff8624127ab8bd" } modify_external_catalogs: true
I20250115 14:52:01.673571 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bb07dc16172246278fff8624127ab8bd" } modify_external_catalogs: true
I20250115 14:52:01.673723 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 5703fc5995184298bf72f9e43efb41f5 (table default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]) successfully deleted
W20250115 14:52:01.673985 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0fd94e4a7fef44f5be462e7a77e8a123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fd94e4a7fef44f5be462e7a77e8a123 already in progress: creating tablet
W20250115 14:52:01.674180 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 104282a09c024bbaafeb5f28f91dadcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104282a09c024bbaafeb5f28f91dadcb already in progress: creating tablet
I20250115 14:52:01.674760 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:01.677852 1168 tablet_bootstrap.cc:654] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:01.680358 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a2fe51b51994491ab38a644bfc55e3ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2fe51b51994491ab38a644bfc55e3ac already in progress: creating tablet
I20250115 14:52:01.680389 1167 ts_tablet_manager.cc:1918] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.680500 1167 log.cc:1198] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f47a8acc3bec46b78725dcc3e675ea74
W20250115 14:52:01.680756 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 496dec75500347699625a755dceed2bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 496dec75500347699625a755dceed2bb already in progress: creating tablet
W20250115 14:52:01.680985 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3f541e916d264738842cd965c7a2a5ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f541e916d264738842cd965c7a2a5ec already in progress: creating tablet
W20250115 14:52:01.682734 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1b2d04209fd34918b83978325a5e1df8 consensus state has no leader
I20250115 14:52:01.684238 1844 raft_consensus.cc:491] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.684391 1844 raft_consensus.cc:513] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.684759 1167 ts_tablet_manager.cc:1939] T f47a8acc3bec46b78725dcc3e675ea74 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:01.685027 1844 leader_election.cc:290] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:01.685477 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 437) took 2224 ms. Trace:
I20250115 14:52:01.685562 1167 rpcz_store.cc:276] 0115 14:51:59.461198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.461242 (+ 44us) service_pool.cc:225] Handling call
0115 14:52:01.475810 (+2014568us) ts_tablet_manager.cc:1145] Deleting tablet f47a8acc3bec46b78725dcc3e675ea74
0115 14:52:01.475819 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.667857 (+192038us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.676983 (+ 9126us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.685461 (+ 8478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2014395}
I20250115 14:52:01.685849 1167 tablet_replica.cc:331] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:01.685804 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92231509c754e8dbfc564813a76a123" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.686030 1167 raft_consensus.cc:2238] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:01.686290 1167 pending_rounds.cc:62] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:01.686295 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f47a8acc3bec46b78725dcc3e675ea74 (table default.Table_aa6a106222694bbfbe6c26b0f8e65396 [id=8725c85d30684db495ec7a20962fd75d]) successfully deleted
I20250115 14:52:01.686326 1167 pending_rounds.cc:69] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558339137368064 op_type: NO_OP noop_request { }
I20250115 14:52:01.686405 1167 raft_consensus.cc:2883] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:01.686468 1167 raft_consensus.cc:2267] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:01.686507 848 leader_election.cc:343] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.686955 1167 ts_tablet_manager.cc:1905] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.692428 1164 tablet_bootstrap.cc:492] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:01.692579 1164 ts_tablet_manager.cc:1397] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.199s user 0.006s sys 0.010s
I20250115 14:52:01.693493 1164 raft_consensus.cc:357] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.693717 1164 raft_consensus.cc:383] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.693811 1164 raft_consensus.cc:738] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.694053 1164 consensus_queue.cc:260] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.694552 1164 ts_tablet_manager.cc:1428] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250115 14:52:01.694523 847 leader_election.cc:343] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b92231509c754e8dbfc564813a76a123
I20250115 14:52:01.694628 847 leader_election.cc:304] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.694816 1164 tablet_bootstrap.cc:492] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:01.696563 1844 raft_consensus.cc:2743] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.697134 1164 tablet_bootstrap.cc:654] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.699242 1168 tablet_bootstrap.cc:492] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:01.699419 1168 ts_tablet_manager.cc:1397] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20250115 14:52:01.700119 1168 raft_consensus.cc:357] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.700376 1168 raft_consensus.cc:383] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.700464 1168 raft_consensus.cc:738] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.700698 1168 consensus_queue.cc:260] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.701313 1167 ts_tablet_manager.cc:1918] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:01.701416 1167 log.cc:1198] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/5703fc5995184298bf72f9e43efb41f5
I20250115 14:52:01.701612 1168 ts_tablet_manager.cc:1428] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:01.701792 1167 ts_tablet_manager.cc:1939] T 5703fc5995184298bf72f9e43efb41f5 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:01.702167 1168 tablet_bootstrap.cc:492] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:01.704267 1168 tablet_bootstrap.cc:654] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.712971 1168 tablet_bootstrap.cc:492] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:01.713091 1168 ts_tablet_manager.cc:1397] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250115 14:52:01.713172 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 438) took 2251 ms. Trace:
I20250115 14:52:01.713292 1167 rpcz_store.cc:276] 0115 14:51:59.461548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.461587 (+ 39us) service_pool.cc:225] Handling call
0115 14:52:01.685804 (+2224217us) ts_tablet_manager.cc:1145] Deleting tablet 5703fc5995184298bf72f9e43efb41f5
0115 14:52:01.685813 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.689763 (+ 3950us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.701224 (+ 11461us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.713117 (+ 11893us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10515,"tablet-delete.queue_time_us":2224067}
I20250115 14:52:01.713809 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 5703fc5995184298bf72f9e43efb41f5 (table default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]) successfully deleted
I20250115 14:52:01.713766 1168 raft_consensus.cc:357] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.713927 1168 raft_consensus.cc:383] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.713979 1168 raft_consensus.cc:738] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.714063 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 439) took 2230 ms. Trace:
I20250115 14:52:01.714154 1168 consensus_queue.cc:260] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.714198 1167 rpcz_store.cc:276] 0115 14:51:59.483907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.484414 (+ 507us) service_pool.cc:225] Handling call
0115 14:52:01.713785 (+2229371us) ts_tablet_manager.cc:1145] Deleting tablet ff9b57b8b9434f55ace5a4f47380df54
0115 14:52:01.713794 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.714008 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2214696}
I20250115 14:52:01.714721 1168 ts_tablet_manager.cc:1428] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:01.715198 1168 tablet_bootstrap.cc:492] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:01.716392 1167 tablet_replica.cc:331] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:01.716928 1168 tablet_bootstrap.cc:654] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.717036 1167 raft_consensus.cc:2238] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.717235 1167 raft_consensus.cc:2267] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.717749 1167 ts_tablet_manager.cc:1905] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.721796 1751 ts_tablet_manager.cc:1918] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:01.724200 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 010e6535b6bc44f391e6f2b6f1686532
I20250115 14:52:01.724720 449 catalog_manager.cc:6596] ReplaceTablet: tablet 010e6535b6bc44f391e6f2b6f1686532 deleted and replaced by tablet 95d891d568574c51b0d062ca9ac77f45
W20250115 14:52:01.726545 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbf94077e0e343b8938b081102dfd2af consensus state has no leader
W20250115 14:52:01.726996 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be63bae96c834a8bbbb1174c027935a2 consensus state has no leader
I20250115 14:52:01.727632 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet fe696560ba6b4a76ad013d9a3486e398
I20250115 14:52:01.727819 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 95718d39a7824517a36c49c3b86fef30
I20250115 14:52:01.728520 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 010e6535b6bc44f391e6f2b6f1686532 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d891d568574c51b0d062ca9ac77f45) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.732192 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 010e6535b6bc44f391e6f2b6f1686532 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d891d568574c51b0d062ca9ac77f45) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.733044 741 tablet_service.cc:1514] Processing DeleteTablet for tablet ac57fa232ae4462793b021b60ede0973 with delete_type TABLET_DATA_DELETED (replaced by tablet cea16f889c73400a85b9553c1975457e) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.734153 879 tablet_service.cc:1514] Processing DeleteTablet for tablet fe696560ba6b4a76ad013d9a3486e398 with delete_type TABLET_DATA_DELETED (replaced by tablet 828be3fd7b634d23b88aebf4633c41fe) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.736955 1166 tablet_replica.cc:331] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:01.737056 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a48227ec5aa846c3ac63bd29ed38be25
I20250115 14:52:01.737159 1166 raft_consensus.cc:2238] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.737370 1166 raft_consensus.cc:2267] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.739540 1166 ts_tablet_manager.cc:1905] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.741472 1168 tablet_bootstrap.cc:492] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:01.741603 1168 ts_tablet_manager.cc:1397] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20250115 14:52:01.742292 1168 raft_consensus.cc:357] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.742461 1168 raft_consensus.cc:383] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.742498 1168 raft_consensus.cc:738] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.742705 1168 consensus_queue.cc:260] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.743300 1168 ts_tablet_manager.cc:1428] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:01.746055 1168 tablet_bootstrap.cc:492] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:01.747189 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d7309ecb11d844848638c042c9bbe5f1 (table default.Table_2e2abefaccf54e469bacf8126edad0c4 [id=855145809143431d81790dd36147a77b]). Expected version 2 got 1
I20250115 14:52:01.723585 1162 tablet_bootstrap.cc:492] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.747380 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 8442dc919d5f4d63b9b2ce7276cc0bc5
I20250115 14:52:01.747454 1751 log.cc:1198] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d4c984613c7d45d5b7c513cd900c9558
I20250115 14:52:01.748383 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 95718d39a7824517a36c49c3b86fef30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.748652 451 catalog_manager.cc:6596] ReplaceTablet: tablet fe696560ba6b4a76ad013d9a3486e398 deleted and replaced by tablet 828be3fd7b634d23b88aebf4633c41fe
I20250115 14:52:01.749244 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:52:01.731600 1728 raft_consensus.cc:491] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.750337 1728 raft_consensus.cc:513] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.750872 453 catalog_manager.cc:6596] ReplaceTablet: tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 deleted and replaced by tablet 8f8efcf1a2db44469cd8dda0b59c34ba
I20250115 14:52:01.750991 1728 leader_election.cc:290] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.751716 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8efcf1a2db44469cd8dda0b59c34ba) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.751927 879 tablet_service.cc:1514] Processing DeleteTablet for tablet a48227ec5aa846c3ac63bd29ed38be25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.751909 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fbde011216426cb41c51fe359dcb5f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.752358 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fbde011216426cb41c51fe359dcb5f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:01.734227 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet fe696560ba6b4a76ad013d9a3486e398 with delete_type TABLET_DATA_DELETED (replaced by tablet 828be3fd7b634d23b88aebf4633c41fe) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.738335 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet a48227ec5aa846c3ac63bd29ed38be25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.747530 1162 ts_tablet_manager.cc:1397] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.217s user 0.024s sys 0.028s
W20250115 14:52:01.753115 983 leader_election.cc:343] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:01.753451 984 leader_election.cc:343] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.729521 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 010e6535b6bc44f391e6f2b6f1686532 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d891d568574c51b0d062ca9ac77f45) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.753571 984 leader_election.cc:304] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.747920 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 95718d39a7824517a36c49c3b86fef30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.753576 1162 raft_consensus.cc:357] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.753938 1162 raft_consensus.cc:383] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.754048 1162 raft_consensus.cc:738] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.754276 1162 consensus_queue.cc:260] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.754711 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8efcf1a2db44469cd8dda0b59c34ba) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.754776 1162 ts_tablet_manager.cc:1428] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:01.754853 1168 tablet_bootstrap.cc:654] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.755081 1162 tablet_bootstrap.cc:492] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.756412 1876 raft_consensus.cc:2743] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.757638 1162 tablet_bootstrap.cc:654] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.758867 741 tablet_service.cc:1514] Processing DeleteTablet for tablet fe696560ba6b4a76ad013d9a3486e398 with delete_type TABLET_DATA_DELETED (replaced by tablet 828be3fd7b634d23b88aebf4633c41fe) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.759486 1750 raft_consensus.cc:491] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.759645 1750 raft_consensus.cc:513] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.760257 1750 leader_election.cc:290] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:01.761622 710 leader_election.cc:343] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 0291acf45d75421b8b1c6311560e7f0c
I20250115 14:52:01.761729 1751 ts_tablet_manager.cc:1939] T d4c984613c7d45d5b7c513cd900c9558 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:01.761838 710 leader_election.cc:343] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 0291acf45d75421b8b1c6311560e7f0c
I20250115 14:52:01.761924 710 leader_election.cc:304] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:01.762300 1750 raft_consensus.cc:2743] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.762480 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 502) took 1479 ms. Trace:
I20250115 14:52:01.763500 741 tablet_service.cc:1514] Processing DeleteTablet for tablet a48227ec5aa846c3ac63bd29ed38be25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.765194 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 95718d39a7824517a36c49c3b86fef30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.767251 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8efcf1a2db44469cd8dda0b59c34ba) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:01.767702 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 010e6535b6bc44f391e6f2b6f1686532 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 010e6535b6bc44f391e6f2b6f1686532 already in progress: creating tablet
I20250115 14:52:01.771867 453 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:52:01.772783 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c5f747dc2f6b4cffb7b791ea6456c5b3" } modify_external_catalogs: true
I20250115 14:52:01.772971 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c5f747dc2f6b4cffb7b791ea6456c5b3" } modify_external_catalogs: true
I20250115 14:52:01.774976 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_12287ed111bb450dbe84bdb433fc746e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.775470 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12287ed111bb450dbe84bdb433fc746e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.777213 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_afd9df7559aa485789c8e7bd5d43c3ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.777710 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afd9df7559aa485789c8e7bd5d43c3ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.781011 1167 ts_tablet_manager.cc:1918] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.781203 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1f3c5d983c50484da1dc4417e5698154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.781602 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f3c5d983c50484da1dc4417e5698154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.782356 1166 ts_tablet_manager.cc:1918] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.782481 1166 log.cc:1198] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ac57fa232ae4462793b021b60ede0973
I20250115 14:52:01.783075 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_600b928b6de9443db6b3c73398af80b7" } new_table_name: "default.Table_25a5a732ec814f24893d09a6f0b38d2c" modify_external_catalogs: true
I20250115 14:52:01.784224 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_98cb517939064897baf89d74e6e0355e" } modify_external_catalogs: true
I20250115 14:52:01.784415 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_98cb517939064897baf89d74e6e0355e" } modify_external_catalogs: true
I20250115 14:52:01.785025 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_903db046a0034d12a7be31cac011cab0" } modify_external_catalogs: true
I20250115 14:52:01.785190 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_903db046a0034d12a7be31cac011cab0" } modify_external_catalogs: true
I20250115 14:52:01.787092 1164 tablet_bootstrap.cc:492] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:01.787268 1164 ts_tablet_manager.cc:1397] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.000s
I20250115 14:52:01.788094 1164 raft_consensus.cc:357] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.788393 1164 raft_consensus.cc:383] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.788514 1164 raft_consensus.cc:738] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.789314 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4d817129c1cf46a6bde83e97a293603b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:01.789131 1164 consensus_queue.cc:260] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:01.789793 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d817129c1cf46a6bde83e97a293603b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.789937 1164 ts_tablet_manager.cc:1428] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:52:01.790971 1164 tablet_bootstrap.cc:492] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:01.767707 1162 tablet_bootstrap.cc:492] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.792621 1162 ts_tablet_manager.cc:1397] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.004s
I20250115 14:52:01.793391 1162 raft_consensus.cc:357] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.793592 1162 raft_consensus.cc:383] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.793663 1162 raft_consensus.cc:738] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.793876 1162 consensus_queue.cc:260] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.794360 1162 ts_tablet_manager.cc:1428] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:01.794656 1162 tablet_bootstrap.cc:492] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.796155 1162 tablet_bootstrap.cc:654] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.799186 1167 log.cc:1198] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/1f1df9dc743342b4b7bdb1a8d3fcd5b0
I20250115 14:52:01.802119 1167 ts_tablet_manager.cc:1939] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:01.803820 1164 tablet_bootstrap.cc:654] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.804407 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 452) took 2057 ms. Trace:
I20250115 14:52:01.792357 1751 rpcz_store.cc:276] 0115 14:52:00.282942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.285826 (+ 2884us) service_pool.cc:225] Handling call
0115 14:52:01.627543 (+1341717us) ts_tablet_manager.cc:1145] Deleting tablet d4c984613c7d45d5b7c513cd900c9558
0115 14:52:01.627555 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.680229 (+ 52674us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.721684 (+ 41455us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.762436 (+ 40752us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341559}
I20250115 14:52:01.804539 1167 rpcz_store.cc:276] 0115 14:51:59.747136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.747192 (+ 56us) service_pool.cc:225] Handling call
0115 14:52:01.715652 (+1968460us) ts_tablet_manager.cc:1145] Deleting tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0
0115 14:52:01.716353 (+ 701us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.731063 (+ 14710us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.780896 (+ 49833us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.804386 (+ 23490us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1968192}
I20250115 14:52:01.804994 1751 tablet_replica.cc:331] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:01.805019 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d4c984613c7d45d5b7c513cd900c9558 (table default.Table_272f9650cb3142f781f1d4d7c4769dfe [id=2fb684a787e4420aa9218b369eba5cf2]) successfully deleted
I20250115 14:52:01.805186 1751 raft_consensus.cc:2238] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:01.805485 1751 raft_consensus.cc:2267] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.805824 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 (table default.Table_fc422b5daf4a4770945aab3d34bc5d35 [id=c8b8a13b708841e2ad2017a4fa7d01d1]) successfully deleted
I20250115 14:52:01.806012 1751 ts_tablet_manager.cc:1905] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.813912 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 453) took 2064 ms. Trace:
I20250115 14:52:01.813933 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_609197916c9b4c81aebdaf70e78cc436 [id=33e3df28f2d445c5ba4e8b0ce4458c1a] alter complete (version 2)
I20250115 14:52:01.814043 1167 rpcz_store.cc:276] 0115 14:51:59.749451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.749497 (+ 46us) service_pool.cc:225] Handling call
0115 14:52:01.813787 (+2064290us) ts_tablet_manager.cc:1145] Deleting tablet 3b82924e735846b2914520684d1b8fc7
0115 14:52:01.813797 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.813897 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2064083}
I20250115 14:52:01.815896 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 454) took 2060 ms. Trace:
I20250115 14:52:01.815963 1167 rpcz_store.cc:276] 0115 14:51:59.755157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.755205 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:01.815792 (+2060587us) ts_tablet_manager.cc:1145] Deleting tablet 5bc975b044dd439dbd1f8f9a82cce751
0115 14:52:01.815802 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.815882 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2060409}
W20250115 14:52:01.816344 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 5bc975b044dd439dbd1f8f9a82cce751 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bc975b044dd439dbd1f8f9a82cce751 already in progress: creating tablet
I20250115 14:52:01.816604 1167 tablet_replica.cc:331] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:01.817456 879 tablet_service.cc:1467] Processing CreateTablet for tablet fab76e0427f14da48bf101db57f0457c (DEFAULT_TABLE table=default.Table_dd66b1c3bf36403bb0d9300ca8f318a5 [id=61e995e8b453479992b8acbb14a05fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.817814 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab76e0427f14da48bf101db57f0457c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.818125 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_607f5ab1bfce4111821d459c200808f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.818660 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_607f5ab1bfce4111821d459c200808f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.821338 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8337d181d44ac7bb82551a677b6831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.821627 882 tablet_service.cc:1467] Processing CreateTablet for tablet b47502309761449dae3be07ea6363238 (DEFAULT_TABLE table=default.Table_f3220f35253d4aa087fd1d48bd8dc8b3 [id=974866671a014ab6b7208c7b5824e8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.821713 1015 tablet_service.cc:1467] Processing CreateTablet for tablet fab76e0427f14da48bf101db57f0457c (DEFAULT_TABLE table=default.Table_dd66b1c3bf36403bb0d9300ca8f318a5 [id=61e995e8b453479992b8acbb14a05fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.821950 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b47502309761449dae3be07ea6363238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.822073 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab76e0427f14da48bf101db57f0457c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.824559 884 tablet_service.cc:1467] Processing CreateTablet for tablet 78a2d39d0296468db48b3eec374244ca (DEFAULT_TABLE table=default.Table_5dd9ae0549364620b988ae28f376f552 [id=4427bf7f81e345b1b5e5a07c81073d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.824884 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a2d39d0296468db48b3eec374244ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.825448 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 78a2d39d0296468db48b3eec374244ca (DEFAULT_TABLE table=default.Table_5dd9ae0549364620b988ae28f376f552 [id=4427bf7f81e345b1b5e5a07c81073d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.825775 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a2d39d0296468db48b3eec374244ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.826040 739 tablet_service.cc:1467] Processing CreateTablet for tablet b47502309761449dae3be07ea6363238 (DEFAULT_TABLE table=default.Table_f3220f35253d4aa087fd1d48bd8dc8b3 [id=974866671a014ab6b7208c7b5824e8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.826381 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b47502309761449dae3be07ea6363238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.828508 1013 tablet_service.cc:1467] Processing CreateTablet for tablet be8e12ccb8d14c1c9258dc3f81d1527d (DEFAULT_TABLE table=default.Table_5a8b86b253c04399a44480d59eeddea5 [id=39c963a5bc8f4a3b9ff5780bb75e33e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.828529 889 tablet_service.cc:1467] Processing CreateTablet for tablet be8e12ccb8d14c1c9258dc3f81d1527d (DEFAULT_TABLE table=default.Table_5a8b86b253c04399a44480d59eeddea5 [id=39c963a5bc8f4a3b9ff5780bb75e33e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.828802 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8e12ccb8d14c1c9258dc3f81d1527d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.828838 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8e12ccb8d14c1c9258dc3f81d1527d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.831993 878 tablet_service.cc:1467] Processing CreateTablet for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa (DEFAULT_TABLE table=default.Table_57cca7a5af0b434b92b6e5c6eb5413f9 [id=cc7f2860ebc141cbae69eea0c8b524f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.832309 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.832563 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b47502309761449dae3be07ea6363238 (DEFAULT_TABLE table=default.Table_f3220f35253d4aa087fd1d48bd8dc8b3 [id=974866671a014ab6b7208c7b5824e8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.832868 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b47502309761449dae3be07ea6363238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.833163 739 tablet_service.cc:1467] Processing CreateTablet for tablet 78a2d39d0296468db48b3eec374244ca (DEFAULT_TABLE table=default.Table_5dd9ae0549364620b988ae28f376f552 [id=4427bf7f81e345b1b5e5a07c81073d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.833456 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78a2d39d0296468db48b3eec374244ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.836560 883 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6ac691e13a4d75be8a42a5d17d9eed (DEFAULT_TABLE table=default.Table_ab69195bec7c448888d88776d4576735 [id=9de5691a38e2454eab629da4839adf20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.836859 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6ac691e13a4d75be8a42a5d17d9eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.839160 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a3a7a487a42949bd9641c38d6c1fcfd3 (table default.Table_8b9b25dd8a85434a83d2641ffbc5189e [id=738170c5571a4d958944ed432428e99d]). Expected version 3 got 0
I20250115 14:52:01.840451 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 0291acf45d75421b8b1c6311560e7f0c with delete_type TABLET_DATA_DELETED (replaced by tablet f90e4edc7aea43268d581b441b788f27) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.840678 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa (DEFAULT_TABLE table=default.Table_57cca7a5af0b434b92b6e5c6eb5413f9 [id=cc7f2860ebc141cbae69eea0c8b524f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.841017 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.841320 749 tablet_service.cc:1467] Processing CreateTablet for tablet be8e12ccb8d14c1c9258dc3f81d1527d (DEFAULT_TABLE table=default.Table_5a8b86b253c04399a44480d59eeddea5 [id=39c963a5bc8f4a3b9ff5780bb75e33e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.841776 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8e12ccb8d14c1c9258dc3f81d1527d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.843945 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e82073df06d449c3870cf6d7b8133825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:01.844452 1728 raft_consensus.cc:491] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:52:01.844507 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e82073df06d449c3870cf6d7b8133825 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.844563 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6ac691e13a4d75be8a42a5d17d9eed (DEFAULT_TABLE table=default.Table_ab69195bec7c448888d88776d4576735 [id=9de5691a38e2454eab629da4839adf20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.844596 1728 raft_consensus.cc:513] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.844895 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6ac691e13a4d75be8a42a5d17d9eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.845283 1728 leader_election.cc:290] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.846105 745 tablet_service.cc:1467] Processing CreateTablet for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa (DEFAULT_TABLE table=default.Table_57cca7a5af0b434b92b6e5c6eb5413f9 [id=cc7f2860ebc141cbae69eea0c8b524f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.846418 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.846575 741 tablet_service.cc:1467] Processing CreateTablet for tablet fab76e0427f14da48bf101db57f0457c (DEFAULT_TABLE table=default.Table_dd66b1c3bf36403bb0d9300ca8f318a5 [id=61e995e8b453479992b8acbb14a05fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.846963 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fab76e0427f14da48bf101db57f0457c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.849589 749 tablet_service.cc:1467] Processing CreateTablet for tablet 2c6ac691e13a4d75be8a42a5d17d9eed (DEFAULT_TABLE table=default.Table_ab69195bec7c448888d88776d4576735 [id=9de5691a38e2454eab629da4839adf20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.849880 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6ac691e13a4d75be8a42a5d17d9eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.850756 875 tablet_service.cc:1514] Processing DeleteTablet for tablet b92231509c754e8dbfc564813a76a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.850965 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 629873c00550408798b00489ece4ea5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.851089 887 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4102b1f83e414d93a6a68315e36d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.851200 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e932d5b820f94130967eccce9e1542fa with delete_type TABLET_DATA_DELETED (replaced by tablet fc3d2ca7cd7c48a79ed1396fbbdc6866) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.851933 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cba5cc51d7414a9352caedcc362c9c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:01.852118 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cba5cc51d7414a9352caedcc362c9c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:01.852615 983 leader_election.cc:343] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:01.854089 984 leader_election.cc:343] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.854207 984 leader_election.cc:304] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.857570 455 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 95d891d568574c51b0d062ca9ac77f45
I20250115 14:52:01.857828 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 4abde784f9c948cb96e6cc93c30817bd from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:01.858237 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 0fd94e4a7fef44f5be462e7a77e8a123 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:01.858762 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0b1e4ccdc6ce444195b8bf86c86ff87b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.859257 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b1e4ccdc6ce444195b8bf86c86ff87b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.860008 1876 raft_consensus.cc:2743] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.863204 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:01.863358 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet caed629878594954842dfc6a40cc8b6d (table default.Table_544d606cf9af4accb670d91d7d717948 [id=b6aeb933c5e64e88a6d794a98650acbe]). Expected version 1 got 0
W20250115 14:52:01.864060 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6942d102069d4c33b23fa2c0b29f94a6 consensus state has no leader
I20250115 14:52:01.864425 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_92acdab3d67543ae8f72d76b7013dbce" } new_table_name: "default.Table_4add05103df84280812bf61910d76d1d" modify_external_catalogs: true
I20250115 14:52:01.865870 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dc6355486b8a4565a2749cf74bec9a2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.866283 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc6355486b8a4565a2749cf74bec9a2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.868258 454 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 1bf55453656f45a8baece81c8aa9ce92
I20250115 14:52:01.869066 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 59d8fd4b26294e8ca6745b95f2e03231
I20250115 14:52:01.871971 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d8fd4b26294e8ca6745b95f2e03231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:01.872190 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d8fd4b26294e8ca6745b95f2e03231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.879371 1728 raft_consensus.cc:491] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.879495 1728 raft_consensus.cc:513] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.880026 1728 leader_election.cc:290] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.880685 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8337d181d44ac7bb82551a677b6831" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:01.881242 984 leader_election.cc:343] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.881791 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8337d181d44ac7bb82551a677b6831" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:01.882266 983 leader_election.cc:343] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.882367 983 leader_election.cc:304] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.882761 1728 raft_consensus.cc:2743] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.892522 1166 ts_tablet_manager.cc:1939] T ac57fa232ae4462793b021b60ede0973 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:01.893530 1166 tablet_replica.cc:331] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:01.893795 1166 raft_consensus.cc:2238] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:01.894249 1166 raft_consensus.cc:2267] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.895056 1166 ts_tablet_manager.cc:1905] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.904439 1167 raft_consensus.cc:2238] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:01.904843 1167 raft_consensus.cc:2267] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:01.908880 1167 ts_tablet_manager.cc:1905] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:01.912984 1168 tablet_bootstrap.cc:492] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:01.913074 1162 tablet_bootstrap.cc:492] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.913105 1168 ts_tablet_manager.cc:1397] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.167s user 0.014s sys 0.015s
I20250115 14:52:01.913204 1162 ts_tablet_manager.cc:1397] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.119s user 0.021s sys 0.021s
I20250115 14:52:01.913763 1168 raft_consensus.cc:357] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.913931 1168 raft_consensus.cc:383] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.913980 1168 raft_consensus.cc:738] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.913880 1162 raft_consensus.cc:357] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.914045 1162 raft_consensus.cc:383] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.914093 1162 raft_consensus.cc:738] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.914146 1168 consensus_queue.cc:260] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.914238 1162 consensus_queue.cc:260] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.914585 1168 ts_tablet_manager.cc:1428] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250115 14:52:01.914844 1168 tablet_bootstrap.cc:492] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:01.914916 1162 ts_tablet_manager.cc:1428] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:01.915561 1162 tablet_bootstrap.cc:492] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.916963 1162 tablet_bootstrap.cc:654] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.917445 1168 tablet_bootstrap.cc:654] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.919081 1164 tablet_bootstrap.cc:492] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:01.919215 1164 ts_tablet_manager.cc:1397] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.128s user 0.001s sys 0.002s
I20250115 14:52:01.919903 1164 raft_consensus.cc:357] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.920090 1164 raft_consensus.cc:383] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.920184 1164 raft_consensus.cc:738] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.920404 1164 consensus_queue.cc:260] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.920697 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d8fd4b26294e8ca6745b95f2e03231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.921586 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ac57fa232ae4462793b021b60ede0973 (table default.Table_fbd024b67b794f99b66433ffbb89184c [id=b3c701a3bba64b0cbedcf0928b3c873c]) successfully deleted
I20250115 14:52:01.922042 1164 ts_tablet_manager.cc:1428] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250115 14:52:01.922901 1164 tablet_bootstrap.cc:492] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:01.925648 889 tablet_service.cc:1514] Processing DeleteTablet for tablet e7100e6a51ed4e178e86138f10d5ffc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6942d102069d4c33b23fa2c0b29f94a6) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.927325 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 83097c5bfe364418a213236435edc40c (table default.Table_24c05b9539414aadad44296c264be5aa [id=caa6adb7563c41088e5120993c46c54f]). Expected version 1 got 0
I20250115 14:52:01.928597 1164 tablet_bootstrap.cc:654] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.932557 1751 ts_tablet_manager.cc:1918] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:01.935199 1164 tablet_bootstrap.cc:492] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:01.935346 1164 ts_tablet_manager.cc:1397] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20250115 14:52:01.936008 1164 raft_consensus.cc:357] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.936650 1751 log.cc:1198] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4239747374aa4d6d866961430786677c
I20250115 14:52:01.937403 1751 ts_tablet_manager.cc:1939] T 4239747374aa4d6d866961430786677c P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:01.936208 1164 raft_consensus.cc:383] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.938726 1164 raft_consensus.cc:738] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.939612 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 503) took 1656 ms. Trace:
I20250115 14:52:01.939862 1164 consensus_queue.cc:260] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.940980 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_24c05b9539414aadad44296c264be5aa" } new_table_name: "default.Table_81803594c5c741619ae708256092f827" modify_external_catalogs: true
I20250115 14:52:01.941728 1164 ts_tablet_manager.cc:1428] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:52:01.942025 1164 tablet_bootstrap.cc:492] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:01.943228 1162 tablet_bootstrap.cc:492] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.942778 1166 ts_tablet_manager.cc:1918] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:01.943353 1162 ts_tablet_manager.cc:1397] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20250115 14:52:01.944007 1162 raft_consensus.cc:357] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.944180 1162 raft_consensus.cc:383] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.944231 1162 raft_consensus.cc:738] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.944428 1162 consensus_queue.cc:260] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.944474 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1fe291920627495bb359337954580862"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:01.944698 1166 log.cc:1198] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/fe696560ba6b4a76ad013d9a3486e398
W20250115 14:52:01.945013 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fe291920627495bb359337954580862 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.945326 1162 ts_tablet_manager.cc:1428] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:01.945700 1164 tablet_bootstrap.cc:654] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.945886 1162 tablet_bootstrap.cc:492] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.947319 1162 tablet_bootstrap.cc:654] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.949384 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a6ee485127484aa71d021edb39ab43 with delete_type TABLET_DATA_DELETED (replaced by tablet 00adfe53c726428b9ac2fcb47756b3de) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.951674 1164 tablet_bootstrap.cc:492] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:01.951817 1164 ts_tablet_manager.cc:1397] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20250115 14:52:01.952983 1164 raft_consensus.cc:357] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.953154 1164 raft_consensus.cc:383] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.953239 1164 raft_consensus.cc:738] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.953459 1164 consensus_queue.cc:260] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.955945 1162 tablet_bootstrap.cc:492] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.956068 1162 ts_tablet_manager.cc:1397] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20250115 14:52:01.956727 1876 raft_consensus.cc:491] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:01.956835 1876 raft_consensus.cc:513] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.957350 1876 leader_election.cc:290] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:01.957655 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f3a87bf8b6452d8afd8dfd836ee4e6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:01.958376 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f3a87bf8b6452d8afd8dfd836ee4e6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:01.958667 1167 ts_tablet_manager.cc:1918] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:01.958783 1167 log.cc:1198] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9f327ffa731c4c949b24a34dc54ae260
W20250115 14:52:01.958961 983 leader_election.cc:343] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.959390 1168 tablet_bootstrap.cc:492] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:01.959498 1168 ts_tablet_manager.cc:1397] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.006s
I20250115 14:52:01.959844 1162 raft_consensus.cc:357] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.960016 1162 raft_consensus.cc:383] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.960076 1162 raft_consensus.cc:738] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
W20250115 14:52:01.960182 984 leader_election.cc:343] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:01.960153 1168 raft_consensus.cc:357] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.960333 1168 raft_consensus.cc:383] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.960381 1168 raft_consensus.cc:738] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.960263 984 leader_election.cc:304] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:01.960578 1168 consensus_queue.cc:260] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.960631 1162 consensus_queue.cc:260] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.960846 1876 raft_consensus.cc:2743] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:01.961138 1168 ts_tablet_manager.cc:1428] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:01.961635 1162 ts_tablet_manager.cc:1428] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250115 14:52:01.961874 1162 tablet_bootstrap.cc:492] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.961903 1168 tablet_bootstrap.cc:492] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:01.963284 1162 tablet_bootstrap.cc:654] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.963572 1168 tablet_bootstrap.cc:654] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.964213 1164 ts_tablet_manager.cc:1428] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20250115 14:52:01.965315 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 51b5bc2c71cf41809d7057acd47e27be with delete_type TABLET_DATA_DELETED (replaced by tablet a300a76b8f7942dea648a20cbb3d5368) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:01.965436 1164 tablet_bootstrap.cc:492] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:01.967260 1164 tablet_bootstrap.cc:654] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.967329 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9c44f30ee706467f8ae6014dd1b51473"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:01.967893 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c44f30ee706467f8ae6014dd1b51473 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:01.970139 1162 tablet_bootstrap.cc:492] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.970252 1162 ts_tablet_manager.cc:1397] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250115 14:52:01.970242 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 276abefcab6a422aa6467f00864c01f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:01.971820 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bccecf4164694596afcaf1710196a41c" } new_table_name: "default.Table_958850c3f0a54c49a8dd42b180068897" modify_external_catalogs: true
I20250115 14:52:01.974412 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_da0cb47ea83b436d9a84f797bc9b02d0" } new_table_name: "default.Table_0034e4c667b44bbb83a5a93e321e5113" modify_external_catalogs: true
I20250115 14:52:01.942806 1751 rpcz_store.cc:276] 0115 14:52:00.283024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.286843 (+ 3819us) service_pool.cc:225] Handling call
0115 14:52:01.804921 (+1518078us) ts_tablet_manager.cc:1145] Deleting tablet 4239747374aa4d6d866961430786677c
0115 14:52:01.804931 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.907743 (+102812us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.932438 (+ 24695us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.939592 (+ 7154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517923}
I20250115 14:52:01.975049 1162 raft_consensus.cc:357] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.975219 1162 raft_consensus.cc:383] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.975275 1162 raft_consensus.cc:738] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.975453 1162 consensus_queue.cc:260] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:01.975008 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 4239747374aa4d6d866961430786677c (table default.Table_8e6652dab3bc46b7b4f1b66415398217 [id=8941c6c27037475990cf318614854a27]) successfully deleted
I20250115 14:52:01.977243 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_61bd6e7ab702444ba0919bfb8efb04f0" } modify_external_catalogs: true
I20250115 14:52:01.977334 1162 ts_tablet_manager.cc:1428] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:52:01.977391 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_61bd6e7ab702444ba0919bfb8efb04f0" } modify_external_catalogs: true
I20250115 14:52:01.977569 1162 tablet_bootstrap.cc:492] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:01.978539 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a1dc2d81847c4f31bc83b06081b8274f" } modify_external_catalogs: true
I20250115 14:52:01.978693 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a1dc2d81847c4f31bc83b06081b8274f" } modify_external_catalogs: true
I20250115 14:52:01.978752 1164 tablet_bootstrap.cc:492] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:01.978892 1164 ts_tablet_manager.cc:1397] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20250115 14:52:01.979104 1162 tablet_bootstrap.cc:654] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.979516 1164 raft_consensus.cc:357] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.979712 1164 raft_consensus.cc:383] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.979787 1164 raft_consensus.cc:738] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.979996 1164 consensus_queue.cc:260] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.980890 1166 ts_tablet_manager.cc:1939] T fe696560ba6b4a76ad013d9a3486e398 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:01.975157 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 489) took 1713 ms. Trace:
I20250115 14:52:01.981067 1751 rpcz_store.cc:276] 0115 14:52:00.262148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.262226 (+ 78us) service_pool.cc:225] Handling call
0115 14:52:01.975039 (+1712813us) ts_tablet_manager.cc:1145] Deleting tablet 4f1ce59e05054a009cd36e12db35d449
0115 14:52:01.975049 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.975138 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1684738}
W20250115 14:52:01.982662 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5265d5eeac83481ca97801e3475886da consensus state has no leader
I20250115 14:52:01.983417 1024 tablet_service.cc:1467] Processing CreateTablet for tablet fa8fbedc053f4373a1ce4267830d6338 (DEFAULT_TABLE table=default.Table_dd281cfe0677416e8a04ee13689d050f [id=7f50ffbf778442b4b754364ccf7e0a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.983773 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa8fbedc053f4373a1ce4267830d6338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.986546 878 tablet_service.cc:1467] Processing CreateTablet for tablet c042a4f4f9bc46afb0b94df8ed74a8cf (DEFAULT_TABLE table=default.Table_02e5510e4edc458bab64cbb11e942275 [id=f35491f7c5474ddc9d88cd0e43cc0047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.986934 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c042a4f4f9bc46afb0b94df8ed74a8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.987924 1162 tablet_bootstrap.cc:492] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:01.988034 1162 ts_tablet_manager.cc:1397] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20250115 14:52:01.988538 883 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8efcf1a2db44469cd8dda0b59c34ba (DEFAULT_TABLE table=default.Table_b3a6b57353484cdf8f2dfde2738ae0c2 [id=5249bd4dd6f14f9bbfced76f1d336c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.988685 1162 raft_consensus.cc:357] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.988864 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8efcf1a2db44469cd8dda0b59c34ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.988868 1162 raft_consensus.cc:383] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:01.988914 1162 raft_consensus.cc:738] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:01.988969 1015 tablet_service.cc:1467] Processing CreateTablet for tablet c574dba6b2a04915bdd871de4dff4df0 (DEFAULT_TABLE table=default.Table_ab10a36d2dd14df0b5b7dde01478a485 [id=30c6c8d04ba34bc2ba1ef39fcdc5c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.989082 1162 consensus_queue.cc:260] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:01.989306 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c574dba6b2a04915bdd871de4dff4df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.991374 1168 tablet_bootstrap.cc:492] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:01.991492 1168 ts_tablet_manager.cc:1397] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20250115 14:52:01.992501 879 tablet_service.cc:1467] Processing CreateTablet for tablet 828be3fd7b634d23b88aebf4633c41fe (DEFAULT_TABLE table=default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.992514 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8efcf1a2db44469cd8dda0b59c34ba (DEFAULT_TABLE table=default.Table_b3a6b57353484cdf8f2dfde2738ae0c2 [id=5249bd4dd6f14f9bbfced76f1d336c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.992774 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 828be3fd7b634d23b88aebf4633c41fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.992789 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8efcf1a2db44469cd8dda0b59c34ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.993201 1164 ts_tablet_manager.cc:1428] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250115 14:52:01.993474 1164 tablet_bootstrap.cc:492] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:01.994998 1164 tablet_bootstrap.cc:654] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:01.996639 1019 tablet_service.cc:1467] Processing CreateTablet for tablet c042a4f4f9bc46afb0b94df8ed74a8cf (DEFAULT_TABLE table=default.Table_02e5510e4edc458bab64cbb11e942275 [id=f35491f7c5474ddc9d88cd0e43cc0047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:01.996946 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c042a4f4f9bc46afb0b94df8ed74a8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:01.997663 1162 ts_tablet_manager.cc:1428] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250115 14:52:01.997888 1162 tablet_bootstrap.cc:492] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.000638 884 tablet_service.cc:1467] Processing CreateTablet for tablet c574dba6b2a04915bdd871de4dff4df0 (DEFAULT_TABLE table=default.Table_ab10a36d2dd14df0b5b7dde01478a485 [id=30c6c8d04ba34bc2ba1ef39fcdc5c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.000949 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c574dba6b2a04915bdd871de4dff4df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.001302 1162 tablet_bootstrap.cc:654] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.004510 889 tablet_service.cc:1467] Processing CreateTablet for tablet fa8fbedc053f4373a1ce4267830d6338 (DEFAULT_TABLE table=default.Table_dd281cfe0677416e8a04ee13689d050f [id=7f50ffbf778442b4b754364ccf7e0a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.004690 1164 tablet_bootstrap.cc:492] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.004812 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa8fbedc053f4373a1ce4267830d6338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.004828 1164 ts_tablet_manager.cc:1397] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20250115 14:52:02.005340 1162 tablet_bootstrap.cc:492] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.005434 1162 ts_tablet_manager.cc:1397] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20250115 14:52:02.005549 1164 raft_consensus.cc:357] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.005751 1164 raft_consensus.cc:383] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.005836 1164 raft_consensus.cc:738] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.006036 1162 raft_consensus.cc:357] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.006052 1164 consensus_queue.cc:260] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.006189 1162 raft_consensus.cc:383] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.006228 1162 raft_consensus.cc:738] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.006393 1162 consensus_queue.cc:260] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.007624 1164 ts_tablet_manager.cc:1428] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:02.007876 1164 tablet_bootstrap.cc:492] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.008735 1162 ts_tablet_manager.cc:1428] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:02.008970 1162 tablet_bootstrap.cc:492] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:02.009553 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 4f1ce59e05054a009cd36e12db35d449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1ce59e05054a009cd36e12db35d449 already in progress: creating tablet
I20250115 14:52:02.009753 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 828be3fd7b634d23b88aebf4633c41fe (DEFAULT_TABLE table=default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.010214 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 828be3fd7b634d23b88aebf4633c41fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.010350 1162 tablet_bootstrap.cc:654] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.010668 1844 raft_consensus.cc:491] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.010789 1844 raft_consensus.cc:513] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.011327 1844 leader_election.cc:290] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.011766 1844 raft_consensus.cc:491] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.011837 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9145bbaf3e234f079e355454f3ce28cb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:02.011868 1844 raft_consensus.cc:513] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.011998 1049 raft_consensus.cc:2463] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:02.012390 847 leader_election.cc:304] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:02.012441 1844 leader_election.cc:290] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.012816 1838 raft_consensus.cc:2798] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:02.012878 1838 raft_consensus.cc:491] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:02.012928 1838 raft_consensus.cc:3054] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:02.013302 847 leader_election.cc:343] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e932d5b820f94130967eccce9e1542fa
I20250115 14:52:02.013818 1164 tablet_bootstrap.cc:654] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.015794 1838 raft_consensus.cc:513] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.016119 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9145bbaf3e234f079e355454f3ce28cb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.017387 1166 tablet_replica.cc:331] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.017391 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9145bbaf3e234f079e355454f3ce28cb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:02.017529 1049 raft_consensus.cc:3054] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.017594 1166 raft_consensus.cc:2238] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.017731 1166 raft_consensus.cc:2267] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.018189 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 504) took 1735 ms. Trace:
I20250115 14:52:02.018260 1751 rpcz_store.cc:276] 0115 14:52:00.283128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.283186 (+ 58us) service_pool.cc:225] Handling call
0115 14:52:02.006228 (+1723042us) ts_tablet_manager.cc:1145] Deleting tablet 3a0d97fa57484b9d8cb879fedda61612
0115 14:52:02.006238 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.018170 (+ 11932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1705072}
I20250115 14:52:02.018635 1751 tablet_replica.cc:331] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.018803 1751 raft_consensus.cc:2238] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:02.019085 1751 raft_consensus.cc:2267] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.019541 1751 ts_tablet_manager.cc:1905] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.019811 1049 raft_consensus.cc:2463] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:02.020267 847 leader_election.cc:304] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:02.022958 1844 raft_consensus.cc:2798] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:02.023073 1844 raft_consensus.cc:695] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:02.023303 1844 consensus_queue.cc:237] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.024636 1164 tablet_bootstrap.cc:492] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.024786 1164 ts_tablet_manager.cc:1397] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.002s
W20250115 14:52:02.025604 848 leader_election.cc:343] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.025405 1164 raft_consensus.cc:357] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.025925 1164 raft_consensus.cc:383] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.026003 1164 raft_consensus.cc:738] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.026221 1164 consensus_queue.cc:260] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.026553 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9145bbaf3e234f079e355454f3ce28cb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:02.026979 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e932d5b820f94130967eccce9e1542fa" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.027828 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 3a0d97fa57484b9d8cb879fedda61612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a0d97fa57484b9d8cb879fedda61612 already in progress: creating tablet
I20250115 14:52:02.027951 1164 ts_tablet_manager.cc:1428] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20250115 14:52:02.028149 848 leader_election.cc:343] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:02.028383 848 leader_election.cc:343] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.028239 1164 tablet_bootstrap.cc:492] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.028520 848 leader_election.cc:304] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.028813 1843 raft_consensus.cc:2743] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.029170 1838 leader_election.cc:290] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.031190 1164 tablet_bootstrap.cc:654] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.032594 1750 raft_consensus.cc:491] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.032761 1750 raft_consensus.cc:513] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.033325 1750 leader_election.cc:290] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.034353 1166 ts_tablet_manager.cc:1905] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:02.034612 710 leader_election.cc:343] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 99a6ee485127484aa71d021edb39ab43
W20250115 14:52:02.034826 710 leader_election.cc:343] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 99a6ee485127484aa71d021edb39ab43
I20250115 14:52:02.034946 710 leader_election.cc:304] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.035372 1750 raft_consensus.cc:2743] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.040043 1164 tablet_bootstrap.cc:492] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.040184 1164 ts_tablet_manager.cc:1397] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20250115 14:52:02.040940 1164 raft_consensus.cc:357] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.041136 1164 raft_consensus.cc:383] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.041217 1164 raft_consensus.cc:738] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.041433 1164 consensus_queue.cc:260] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.042908 1164 ts_tablet_manager.cc:1428] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250115 14:52:02.043179 1164 tablet_bootstrap.cc:492] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.045347 1162 tablet_bootstrap.cc:492] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.045490 1162 ts_tablet_manager.cc:1397] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.007s
I20250115 14:52:02.045821 1164 tablet_bootstrap.cc:654] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.046180 1162 raft_consensus.cc:357] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.046352 1162 raft_consensus.cc:383] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.046407 1162 raft_consensus.cc:738] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.046584 1162 consensus_queue.cc:260] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:02.046842 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:40227, user_credentials={real_user=slave}} blocked reactor thread for 67446.1us
I20250115 14:52:02.051756 1162 ts_tablet_manager.cc:1428] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:52:02.052026 1162 tablet_bootstrap.cc:492] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.052044 1750 raft_consensus.cc:491] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.052202 1750 raft_consensus.cc:513] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.052829 1750 leader_election.cc:290] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.053489 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f847f92cd122423aa93f1afa701b0bbb" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.053690 913 raft_consensus.cc:2463] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
W20250115 14:52:02.053807 710 leader_election.cc:343] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: f847f92cd122423aa93f1afa701b0bbb
I20250115 14:52:02.054062 710 leader_election.cc:304] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.054554 1750 raft_consensus.cc:2798] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:02.054639 1750 raft_consensus.cc:491] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:02.054750 1750 raft_consensus.cc:3054] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.054939 1164 tablet_bootstrap.cc:492] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.055097 1164 ts_tablet_manager.cc:1397] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20250115 14:52:02.055832 1164 raft_consensus.cc:357] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.056056 1164 raft_consensus.cc:383] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.056126 1164 raft_consensus.cc:738] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
W20250115 14:52:02.056161 848 consensus_peers.cc:487] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:02.056596 1162 tablet_bootstrap.cc:654] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.056675 1164 consensus_queue.cc:260] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.057299 1167 ts_tablet_manager.cc:1939] T 9f327ffa731c4c949b24a34dc54ae260 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:02.058198 741 tablet_service.cc:1467] Processing CreateTablet for tablet fa8fbedc053f4373a1ce4267830d6338 (DEFAULT_TABLE table=default.Table_dd281cfe0677416e8a04ee13689d050f [id=7f50ffbf778442b4b754364ccf7e0a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.058557 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa8fbedc053f4373a1ce4267830d6338. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.058667 1750 raft_consensus.cc:513] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.059382 1750 leader_election.cc:290] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.062744 1164 ts_tablet_manager.cc:1428] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250115 14:52:02.063047 1164 tablet_bootstrap.cc:492] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.065017 1162 tablet_bootstrap.cc:492] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.065140 1162 ts_tablet_manager.cc:1397] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250115 14:52:02.065822 1162 raft_consensus.cc:357] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.066013 1162 raft_consensus.cc:383] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.066069 1162 raft_consensus.cc:738] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
W20250115 14:52:02.066061 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 1b2d04209fd34918b83978325a5e1df8 (table default.Table_d548be616a2c4f01ac52eadd5c009eaf [id=1907295f0b5c49b9b99a188d4dc9f7f8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1b2d04209fd34918b83978325a5e1df8 consensus state has no leader
I20250115 14:52:02.066244 1164 tablet_bootstrap.cc:654] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.066236 1162 consensus_queue.cc:260] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:02.067296 710 leader_election.cc:343] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: f847f92cd122423aa93f1afa701b0bbb
I20250115 14:52:02.066964 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f847f92cd122423aa93f1afa701b0bbb" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:02.067898 913 raft_consensus.cc:3054] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.070679 913 raft_consensus.cc:2463] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:02.071235 710 leader_election.cc:304] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.071578 1955 raft_consensus.cc:2798] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:02.071754 1955 raft_consensus.cc:695] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:02.072160 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 455) took 2224 ms. Trace:
I20250115 14:52:02.072311 1955 consensus_queue.cc:237] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.073395 1751 ts_tablet_manager.cc:1918] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:02.073715 1751 log.cc:1198] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d5a5a92804cb462181e68e1cec02102b
I20250115 14:52:02.073549 741 tablet_service.cc:1467] Processing CreateTablet for tablet c042a4f4f9bc46afb0b94df8ed74a8cf (DEFAULT_TABLE table=default.Table_02e5510e4edc458bab64cbb11e942275 [id=f35491f7c5474ddc9d88cd0e43cc0047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.074218 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c042a4f4f9bc46afb0b94df8ed74a8cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.074735 1162 ts_tablet_manager.cc:1428] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250115 14:52:02.076668 1751 ts_tablet_manager.cc:1939] T d5a5a92804cb462181e68e1cec02102b P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.077024 1162 tablet_bootstrap.cc:492] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.077138 745 tablet_service.cc:1467] Processing CreateTablet for tablet c574dba6b2a04915bdd871de4dff4df0 (DEFAULT_TABLE table=default.Table_ab10a36d2dd14df0b5b7dde01478a485 [id=30c6c8d04ba34bc2ba1ef39fcdc5c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.077383 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 501) took 1794 ms. Trace:
I20250115 14:52:02.077476 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c574dba6b2a04915bdd871de4dff4df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.077452 1751 rpcz_store.cc:276] 0115 14:52:00.282852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.285753 (+ 2901us) service_pool.cc:225] Handling call
0115 14:52:02.018593 (+1732840us) ts_tablet_manager.cc:1145] Deleting tablet d5a5a92804cb462181e68e1cec02102b
0115 14:52:02.018602 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.022347 (+ 3745us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.072486 (+ 50139us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.077368 (+ 4882us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1715346}
I20250115 14:52:02.079114 1162 tablet_bootstrap.cc:654] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.079352 1166 ts_tablet_manager.cc:1918] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.079486 1166 log.cc:1198] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/0291acf45d75421b8b1c6311560e7f0c
I20250115 14:52:02.079833 1166 ts_tablet_manager.cc:1939] T 0291acf45d75421b8b1c6311560e7f0c P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:02.081250 1751 tablet_replica.cc:331] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.081481 1751 raft_consensus.cc:2238] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.081704 1751 raft_consensus.cc:2267] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.084554 1751 ts_tablet_manager.cc:1905] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.086438 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d5a5a92804cb462181e68e1cec02102b (table default.Table_c0256da787104a42a1bc1e3946e11fd6 [id=22c418c3ac6648c687a8ae14d439aea5]) successfully deleted
I20250115 14:52:02.084318 1838 raft_consensus.cc:491] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.087147 1838 raft_consensus.cc:513] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.087680 1166 tablet_replica.cc:331] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.087878 1166 raft_consensus.cc:2238] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.088114 1838 leader_election.cc:290] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:02.088814 847 leader_election.cc:343] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: bd4102b1f83e414d93a6a68315e36d16
I20250115 14:52:02.090478 1166 raft_consensus.cc:2267] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.090584 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4102b1f83e414d93a6a68315e36d16" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.090845 776 raft_consensus.cc:2463] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:02.091708 1166 ts_tablet_manager.cc:1905] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:02.094273 847 consensus_peers.cc:487] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:02.094841 848 leader_election.cc:304] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.095278 1838 raft_consensus.cc:2798] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:02.095399 1838 raft_consensus.cc:491] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:02.095506 1838 raft_consensus.cc:3054] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.098096 1838 raft_consensus.cc:513] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.099295 1751 ts_tablet_manager.cc:1918] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250115 14:52:02.099521 847 leader_election.cc:343] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: bd4102b1f83e414d93a6a68315e36d16
I20250115 14:52:02.099758 1167 rpcz_store.cc:276] 0115 14:51:59.847477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.847536 (+ 59us) service_pool.cc:225] Handling call
0115 14:52:01.816560 (+1969024us) ts_tablet_manager.cc:1145] Deleting tablet 9f327ffa731c4c949b24a34dc54ae260
0115 14:52:01.816569 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:01.911859 (+ 95290us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:01.958576 (+ 46717us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.072145 (+113569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1968756}
I20250115 14:52:02.100610 1751 log.cc:1198] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4b90205ef15f470a99c013e7f2cf99c6
I20250115 14:52:02.100561 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9f327ffa731c4c949b24a34dc54ae260 (table default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]) successfully deleted
I20250115 14:52:02.100121 1838 leader_election.cc:290] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.103026 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 456) took 2238 ms. Trace:
I20250115 14:52:02.105742 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet fe696560ba6b4a76ad013d9a3486e398 (table default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]) successfully deleted
I20250115 14:52:02.106914 1167 rpcz_store.cc:276] 0115 14:51:59.864584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.876261 (+ 11677us) service_pool.cc:225] Handling call
0115 14:52:02.102916 (+2226655us) ts_tablet_manager.cc:1145] Deleting tablet 248cadcdef8b4e99923cdbca39690685
0115 14:52:02.102927 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.103008 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2226352}
W20250115 14:52:02.107525 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a48227ec5aa846c3ac63bd29ed38be25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48227ec5aa846c3ac63bd29ed38be25 already in progress: creating tablet
I20250115 14:52:02.102957 1168 raft_consensus.cc:357] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.108115 1168 raft_consensus.cc:383] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.108191 1168 raft_consensus.cc:738] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.108405 1168 consensus_queue.cc:260] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.108774 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 457) took 2179 ms. Trace:
W20250115 14:52:02.107933 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 248cadcdef8b4e99923cdbca39690685 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 248cadcdef8b4e99923cdbca39690685 already in progress: creating tablet
W20250115 14:52:02.109196 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 95718d39a7824517a36c49c3b86fef30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95718d39a7824517a36c49c3b86fef30 already in progress: creating tablet
I20250115 14:52:02.108940 1168 ts_tablet_manager.cc:1428] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.117s user 0.002s sys 0.000s
I20250115 14:52:02.109630 1168 tablet_bootstrap.cc:492] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:02.111318 1168 tablet_bootstrap.cc:654] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.111366 1751 ts_tablet_manager.cc:1939] T 4b90205ef15f470a99c013e7f2cf99c6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.115746 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 511) took 1831 ms. Trace:
I20250115 14:52:02.116636 1751 rpcz_store.cc:276] 0115 14:52:00.284043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.287007 (+ 2964us) service_pool.cc:225] Handling call
0115 14:52:02.077706 (+1790699us) ts_tablet_manager.cc:1145] Deleting tablet 4b90205ef15f470a99c013e7f2cf99c6
0115 14:52:02.077949 (+ 243us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.088790 (+ 10841us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.099211 (+ 10421us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.115724 (+ 16513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1774158}
I20250115 14:52:02.117182 1876 raft_consensus.cc:491] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.117367 1876 raft_consensus.cc:513] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.117933 1876 leader_election.cc:290] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.118608 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d195be2d543f4a8693045a024912c43a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:02.119180 983 leader_election.cc:343] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.119787 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 527) took 1344 ms. Trace:
I20250115 14:52:02.120193 1751 rpcz_store.cc:276] 0115 14:52:00.775159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.885515 (+110356us) service_pool.cc:225] Handling call
0115 14:52:02.119678 (+1234163us) ts_tablet_manager.cc:1145] Deleting tablet 4f89ec0a5d68408c897f43ab39df62b1
0115 14:52:02.119688 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.119772 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210117}
I20250115 14:52:02.120807 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 526) took 1345 ms. Trace:
I20250115 14:52:02.120731 1167 rpcz_store.cc:276] 0115 14:51:59.928614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.928847 (+ 233us) service_pool.cc:225] Handling call
0115 14:52:02.108445 (+2179598us) ts_tablet_manager.cc:1145] Deleting tablet 4294c49da1e044be993100b3cf5e2cea
0115 14:52:02.108457 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.108546 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2179364}
I20250115 14:52:02.121002 1751 rpcz_store.cc:276] 0115 14:52:00.775027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.884491 (+109464us) service_pool.cc:225] Handling call
0115 14:52:02.120687 (+1236196us) ts_tablet_manager.cc:1145] Deleting tablet 4adf1353bd554d8294a7f570931095d2
0115 14:52:02.120698 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.120789 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155979}
I20250115 14:52:02.121183 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 458) took 2192 ms. Trace:
I20250115 14:52:02.121312 1167 rpcz_store.cc:276] 0115 14:51:59.928793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:51:59.928896 (+ 103us) service_pool.cc:225] Handling call
0115 14:52:02.121072 (+2192176us) ts_tablet_manager.cc:1145] Deleting tablet 6e616d78933e4f84b9ec2acc51450b18
0115 14:52:02.121081 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.121169 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2191978}
I20250115 14:52:02.121836 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 459) took 2113 ms. Trace:
W20250115 14:52:02.122637 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 already in progress: creating tablet
I20250115 14:52:02.123167 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 0291acf45d75421b8b1c6311560e7f0c (table default.Table_bbe00fc656b44633ae2fd159ae77ffc0 [id=e6f49fcfb7d74c71904b4e6b1bdfee1d]) successfully deleted
W20250115 14:52:02.123482 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 59d8fd4b26294e8ca6745b95f2e03231 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59d8fd4b26294e8ca6745b95f2e03231 already in progress: creating tablet
I20250115 14:52:02.122162 1167 rpcz_store.cc:276] 0115 14:52:00.008882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.008951 (+ 69us) service_pool.cc:225] Handling call
0115 14:52:02.121679 (+2112728us) ts_tablet_manager.cc:1145] Deleting tablet 43605c995df4400e91139199c9514bfd
0115 14:52:02.121712 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.121818 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2112493}
I20250115 14:52:02.127707 1167 tablet_replica.cc:331] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:02.127912 1167 raft_consensus.cc:2238] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.128098 1167 raft_consensus.cc:2267] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.128448 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 534) took 1352 ms. Trace:
I20250115 14:52:02.128644 1167 ts_tablet_manager.cc:1905] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.128576 1751 rpcz_store.cc:276] 0115 14:52:00.775725 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.964745 (+189020us) service_pool.cc:225] Handling call
0115 14:52:02.128335 (+1163590us) ts_tablet_manager.cc:1145] Deleting tablet e034d7761a5247c79414b7112a1c97b2
0115 14:52:02.128345 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.128429 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138673}
I20250115 14:52:02.119529 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 4b90205ef15f470a99c013e7f2cf99c6 (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]) successfully deleted
W20250115 14:52:02.129279 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 4294c49da1e044be993100b3cf5e2cea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4294c49da1e044be993100b3cf5e2cea already in progress: creating tablet
W20250115 14:52:02.129501 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6e616d78933e4f84b9ec2acc51450b18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e616d78933e4f84b9ec2acc51450b18 already in progress: creating tablet
W20250115 14:52:02.129726 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 43605c995df4400e91139199c9514bfd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43605c995df4400e91139199c9514bfd already in progress: creating tablet
W20250115 14:52:02.129916 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 4f89ec0a5d68408c897f43ab39df62b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f89ec0a5d68408c897f43ab39df62b1 already in progress: creating tablet
W20250115 14:52:02.130107 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 4adf1353bd554d8294a7f570931095d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4adf1353bd554d8294a7f570931095d2 already in progress: creating tablet
W20250115 14:52:02.130306 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e034d7761a5247c79414b7112a1c97b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e034d7761a5247c79414b7112a1c97b2 already in progress: creating tablet
I20250115 14:52:02.128957 1751 tablet_replica.cc:331] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.130745 1751 raft_consensus.cc:2238] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.130890 1751 raft_consensus.cc:2267] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.132105 1751 ts_tablet_manager.cc:1905] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.134868 739 tablet_service.cc:1467] Processing CreateTablet for tablet 828be3fd7b634d23b88aebf4633c41fe (DEFAULT_TABLE table=default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.135403 745 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8efcf1a2db44469cd8dda0b59c34ba (DEFAULT_TABLE table=default.Table_b3a6b57353484cdf8f2dfde2738ae0c2 [id=5249bd4dd6f14f9bbfced76f1d336c71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.135721 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d195be2d543f4a8693045a024912c43a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.136220 984 leader_election.cc:343] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.136353 984 leader_election.cc:304] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.136687 1876 raft_consensus.cc:2743] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.137485 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 828be3fd7b634d23b88aebf4633c41fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.138281 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f327ffa731c4c949b24a34dc54ae260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.139036 1164 tablet_bootstrap.cc:492] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.139196 1164 ts_tablet_manager.cc:1397] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.003s
I20250115 14:52:02.139772 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4102b1f83e414d93a6a68315e36d16" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:02.141614 772 raft_consensus.cc:3054] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.151121 1164 raft_consensus.cc:357] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.155922 1164 raft_consensus.cc:383] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.156783 1164 raft_consensus.cc:738] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.157032 1164 consensus_queue.cc:260] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.157861 1164 ts_tablet_manager.cc:1428] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250115 14:52:02.158979 1164 tablet_bootstrap.cc:492] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.160177 772 raft_consensus.cc:2463] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
W20250115 14:52:02.162943 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa57c1fa96594a40b2551ad503127e4f consensus state has no leader
I20250115 14:52:02.172323 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f8efcf1a2db44469cd8dda0b59c34ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.175410 1164 tablet_bootstrap.cc:654] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.164702 848 leader_election.cc:304] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.180744 1838 raft_consensus.cc:2798] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:02.180874 1838 raft_consensus.cc:695] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:02.181098 1838 consensus_queue.cc:237] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.182096 1843 raft_consensus.cc:491] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.182201 1843 raft_consensus.cc:513] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.182723 1843 leader_election.cc:290] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:02.183688 847 leader_election.cc:343] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 629873c00550408798b00489ece4ea5f
I20250115 14:52:02.184185 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_854a113be33b4b698a50ae5bd10a2fff" } modify_external_catalogs: true
I20250115 14:52:02.184440 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_854a113be33b4b698a50ae5bd10a2fff" } modify_external_catalogs: true
I20250115 14:52:02.185277 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c699f7ff601546bb985cc2e19af86edb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.185905 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c699f7ff601546bb985cc2e19af86edb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.187153 1728 raft_consensus.cc:491] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.187279 1728 raft_consensus.cc:513] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.187855 1728 leader_election.cc:290] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.189739 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd41d42331494e9c1b021816d0287e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:02.190371 983 leader_election.cc:343] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.192385 1166 ts_tablet_manager.cc:1918] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.192390 1751 ts_tablet_manager.cc:1918] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.192549 1166 log.cc:1198] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/99a6ee485127484aa71d021edb39ab43
I20250115 14:52:02.192703 1751 log.cc:1198] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9145bbaf3e234f079e355454f3ce28cb
I20250115 14:52:02.192720 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd41d42331494e9c1b021816d0287e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.192961 1166 ts_tablet_manager.cc:1939] T 99a6ee485127484aa71d021edb39ab43 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:02.193204 1751 ts_tablet_manager.cc:1939] T 9145bbaf3e234f079e355454f3ce28cb P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:02.193331 984 leader_election.cc:343] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.193459 984 leader_election.cc:304] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.193686 1876 raft_consensus.cc:2743] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.193852 1166 tablet_replica.cc:331] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.194067 1166 raft_consensus.cc:2238] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.194231 1166 raft_consensus.cc:2267] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.195086 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 535) took 1419 ms. Trace:
I20250115 14:52:02.195248 1751 rpcz_store.cc:276] 0115 14:52:00.775805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.992719 (+216914us) service_pool.cc:225] Handling call
0115 14:52:02.128914 (+1136195us) ts_tablet_manager.cc:1145] Deleting tablet 9145bbaf3e234f079e355454f3ce28cb
0115 14:52:02.128923 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.140624 (+ 11701us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.192259 (+ 51635us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.195048 (+ 2789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1135782}
I20250115 14:52:02.195888 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9145bbaf3e234f079e355454f3ce28cb (table default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]) successfully deleted
I20250115 14:52:02.196455 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 536) took 1420 ms. Trace:
I20250115 14:52:02.196611 1751 rpcz_store.cc:276] 0115 14:52:00.775893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.993186 (+217293us) service_pool.cc:225] Handling call
0115 14:52:02.196249 (+1203063us) ts_tablet_manager.cc:1145] Deleting tablet 9f4475441e4a467a8d593b742d2a5b80
0115 14:52:02.196278 (+ 29us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.196436 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1202776}
W20250115 14:52:02.197423 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9f4475441e4a467a8d593b742d2a5b80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f4475441e4a467a8d593b742d2a5b80 already in progress: creating tablet
I20250115 14:52:02.197800 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "629873c00550408798b00489ece4ea5f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.198495 848 leader_election.cc:343] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.198721 848 leader_election.cc:304] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.199579 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 542) took 1377 ms. Trace:
I20250115 14:52:02.199749 1751 rpcz_store.cc:276] 0115 14:52:00.822083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.044177 (+222094us) service_pool.cc:225] Handling call
0115 14:52:02.199430 (+1155253us) ts_tablet_manager.cc:1145] Deleting tablet 759a4ebe7b0444128ddd31a78aad463e
0115 14:52:02.199440 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.199562 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1106129}
W20250115 14:52:02.200336 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 759a4ebe7b0444128ddd31a78aad463e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 759a4ebe7b0444128ddd31a78aad463e already in progress: creating tablet
I20250115 14:52:02.200785 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 554) took 1228 ms. Trace:
I20250115 14:52:02.200953 1751 rpcz_store.cc:276] 0115 14:52:00.983231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.227991 (+244760us) service_pool.cc:225] Handling call
0115 14:52:02.200647 (+972656us) ts_tablet_manager.cc:1145] Deleting tablet 6308f68e41554470b7d1321c60d4eee9
0115 14:52:02.200656 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.200770 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":968857}
W20250115 14:52:02.201445 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6308f68e41554470b7d1321c60d4eee9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6308f68e41554470b7d1321c60d4eee9 already in progress: creating tablet
I20250115 14:52:02.201879 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 553) took 1240 ms. Trace:
I20250115 14:52:02.202049 1751 rpcz_store.cc:276] 0115 14:52:00.961922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.210278 (+248356us) service_pool.cc:225] Handling call
0115 14:52:02.201741 (+991463us) ts_tablet_manager.cc:1145] Deleting tablet 0cae4c9228c640e495507e8f7ad9f544
0115 14:52:02.201750 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.201863 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":968614}
W20250115 14:52:02.202553 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0cae4c9228c640e495507e8f7ad9f544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cae4c9228c640e495507e8f7ad9f544 already in progress: creating tablet
I20250115 14:52:02.203033 1843 raft_consensus.cc:2743] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.203020 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 555) took 1219 ms. Trace:
I20250115 14:52:02.203107 1751 rpcz_store.cc:276] 0115 14:52:00.983440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.231042 (+247602us) service_pool.cc:225] Handling call
0115 14:52:02.202851 (+971809us) ts_tablet_manager.cc:1145] Deleting tablet b5177265ba974ad6a381adb1e8c49589
0115 14:52:02.202859 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.202977 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":963026}
W20250115 14:52:02.203513 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b5177265ba974ad6a381adb1e8c49589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5177265ba974ad6a381adb1e8c49589 already in progress: creating tablet
W20250115 14:52:02.204341 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0fd94e4a7fef44f5be462e7a77e8a123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fd94e4a7fef44f5be462e7a77e8a123 already in progress: creating tablet
I20250115 14:52:02.204481 1843 consensus_queue.cc:1035] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:52:02.204974 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 104282a09c024bbaafeb5f28f91dadcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104282a09c024bbaafeb5f28f91dadcb already in progress: creating tablet
W20250115 14:52:02.205657 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9f327ffa731c4c949b24a34dc54ae260 because the tablet was not found. No further retry: Not found: Tablet not found: 9f327ffa731c4c949b24a34dc54ae260
I20250115 14:52:02.206017 1751 tablet_replica.cc:331] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.206326 1751 raft_consensus.cc:2238] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.206549 1751 raft_consensus.cc:2267] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.207340 1751 ts_tablet_manager.cc:1905] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.211083 1166 ts_tablet_manager.cc:1905] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:02.214460 847 consensus_peers.cc:487] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0de6f6f014074c3d8775cb5e4121ccb0. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:02.216933 1751 ts_tablet_manager.cc:1918] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.217979 1751 log.cc:1198] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/276abefcab6a422aa6467f00864c01f4
I20250115 14:52:02.218538 1751 ts_tablet_manager.cc:1939] T 276abefcab6a422aa6467f00864c01f4 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.221899 1168 tablet_bootstrap.cc:492] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:02.222088 1168 ts_tablet_manager.cc:1397] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.005s
I20250115 14:52:02.222923 1168 raft_consensus.cc:357] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.223115 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 276abefcab6a422aa6467f00864c01f4 (table default.Table_c3fec71fedac4b4c9864b18ea6af10b9 [id=382fc98d4c614a9fb79d50a9d630b36d]) successfully deleted
I20250115 14:52:02.223181 1168 raft_consensus.cc:383] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.223232 1168 raft_consensus.cc:738] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.223438 1168 consensus_queue.cc:260] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.224146 1168 ts_tablet_manager.cc:1428] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250115 14:52:02.224354 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 3f541e916d264738842cd965c7a2a5ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f541e916d264738842cd965c7a2a5ec already in progress: creating tablet
I20250115 14:52:02.224747 1751 tablet_replica.cc:331] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.225008 1168 tablet_bootstrap.cc:492] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:02.225046 1751 raft_consensus.cc:2238] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.225344 1751 raft_consensus.cc:2267] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.226969 1168 tablet_bootstrap.cc:654] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.228767 1751 ts_tablet_manager.cc:1905] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.233907 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b8b42fde86054733ac70ea83d43d6f1b" } new_table_name: "default.Table_65de108282934cad838f2c1f716989a5" modify_external_catalogs: true
I20250115 14:52:02.234400 1166 ts_tablet_manager.cc:1918] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.234545 1166 log.cc:1198] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/51b5bc2c71cf41809d7057acd47e27be
I20250115 14:52:02.234884 1166 ts_tablet_manager.cc:1939] T 51b5bc2c71cf41809d7057acd47e27be P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:02.235785 1166 tablet_replica.cc:331] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.235982 1166 raft_consensus.cc:2238] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.237696 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bbcbd9770e46481a894a5352459d371d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:02.237967 1164 tablet_bootstrap.cc:492] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.238118 1164 ts_tablet_manager.cc:1397] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.008s
W20250115 14:52:02.238243 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbcbd9770e46481a894a5352459d371d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.238819 1164 raft_consensus.cc:357] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.239029 1164 raft_consensus.cc:383] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.239101 1164 raft_consensus.cc:738] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.239365 1164 consensus_queue.cc:260] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.239967 1164 ts_tablet_manager.cc:1428] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:02.240504 1751 ts_tablet_manager.cc:1918] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.240725 1751 log.cc:1198] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/496dec75500347699625a755dceed2bb
I20250115 14:52:02.241137 1751 ts_tablet_manager.cc:1939] T 496dec75500347699625a755dceed2bb P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.241971 1164 tablet_bootstrap.cc:492] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.242934 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 496dec75500347699625a755dceed2bb (table default.Table_cb0308c238d7479dbcc041fe3d6e7eff [id=983e866df38543b0b2502165df44b94c]) successfully deleted
W20250115 14:52:02.243772 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a2fe51b51994491ab38a644bfc55e3ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2fe51b51994491ab38a644bfc55e3ac already in progress: creating tablet
I20250115 14:52:02.244180 1751 tablet_replica.cc:331] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.244480 1751 raft_consensus.cc:2238] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.244740 1164 tablet_bootstrap.cc:654] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.244755 1751 raft_consensus.cc:2267] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.245577 1166 raft_consensus.cc:2267] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.245600 1751 ts_tablet_manager.cc:1905] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.246239 1166 ts_tablet_manager.cc:1905] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.254065 1751 ts_tablet_manager.cc:1918] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.254245 1751 log.cc:1198] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e3fbde011216426cb41c51fe359dcb5f
I20250115 14:52:02.254722 1751 ts_tablet_manager.cc:1939] T e3fbde011216426cb41c51fe359dcb5f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.255976 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e3fbde011216426cb41c51fe359dcb5f (table default.Table_1cd00e265b464348bd1df215a1a0d9e9 [id=3fe917c536db4535b79e95a59f9ea8f8]) successfully deleted
I20250115 14:52:02.256393 1751 tablet_replica.cc:331] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.256667 1751 raft_consensus.cc:2238] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.256914 1751 raft_consensus.cc:2267] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.257048 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ec781cbe91bf4224b2b7c28a98ccac67" } modify_external_catalogs: true
I20250115 14:52:02.257236 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ec781cbe91bf4224b2b7c28a98ccac67" } modify_external_catalogs: true
I20250115 14:52:02.258383 1751 ts_tablet_manager.cc:1905] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.261081 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bf35b2839ef143e4b6aeec7caf9313ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.261642 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf35b2839ef143e4b6aeec7caf9313ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.269496 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 99a6ee485127484aa71d021edb39ab43 (table default.Table_d3300e90dc3845f78b3c63dfbf681dde [id=cba9cd7a6d434f18b9e621b035a3a6e0]) successfully deleted
I20250115 14:52:02.269686 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 51b5bc2c71cf41809d7057acd47e27be (table default.Table_da0cb47ea83b436d9a84f797bc9b02d0 [id=f6bae7686ebc45488f5eb27a554d28a3]) successfully deleted
I20250115 14:52:02.271196 1164 tablet_bootstrap.cc:492] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.271423 1164 ts_tablet_manager.cc:1397] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20250115 14:52:02.272506 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a
I20250115 14:52:02.272492 1164 raft_consensus.cc:357] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.272881 1164 raft_consensus.cc:383] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.273018 1164 raft_consensus.cc:738] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.273326 1164 consensus_queue.cc:260] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.274077 1164 ts_tablet_manager.cc:1428] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:02.274411 1164 tablet_bootstrap.cc:492] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.274688 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:52:02.276530 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e6978223c1984a38bec5f2d06a9cf457
I20250115 14:52:02.276686 1164 tablet_bootstrap.cc:654] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.277184 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9af73c3d469c4f2099396a0d51abc8ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.277740 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9af73c3d469c4f2099396a0d51abc8ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.279613 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.279613 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet e6978223c1984a38bec5f2d06a9cf457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.286686 1164 tablet_bootstrap.cc:492] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.286947 1164 ts_tablet_manager.cc:1397] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20250115 14:52:02.288125 1164 raft_consensus.cc:357] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.288465 1164 raft_consensus.cc:383] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.288594 1164 raft_consensus.cc:738] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
W20250115 14:52:02.288859 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a300a76b8f7942dea648a20cbb3d5368 consensus state has no leader
I20250115 14:52:02.289054 1164 consensus_queue.cc:260] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.289361 1751 ts_tablet_manager.cc:1918] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.289244 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_25a5a732ec814f24893d09a6f0b38d2c" } modify_external_catalogs: true
I20250115 14:52:02.290020 1955 raft_consensus.cc:491] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.290165 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_25a5a732ec814f24893d09a6f0b38d2c" } modify_external_catalogs: true
I20250115 14:52:02.290197 1955 raft_consensus.cc:513] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.290773 1955 leader_election.cc:290] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.291404 1166 ts_tablet_manager.cc:1918] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.291523 1166 log.cc:1198] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9f327ffa731c4c949b24a34dc54ae260
I20250115 14:52:02.291903 1166 ts_tablet_manager.cc:1939] T 9f327ffa731c4c949b24a34dc54ae260 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:52:02.291908 710 leader_election.cc:343] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e932d5b820f94130967eccce9e1542fa
I20250115 14:52:02.292179 1751 log.cc:1198] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/23cba5cc51d7414a9352caedcc362c9c
I20250115 14:52:02.292474 1751 ts_tablet_manager.cc:1939] T 23cba5cc51d7414a9352caedcc362c9c P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.292613 1164 ts_tablet_manager.cc:1428] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:52:02.294353 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 23cba5cc51d7414a9352caedcc362c9c (table default.Table_27cd4d793c8f4e77aa531c06dfa74a2d [id=a6994f260e1a4eada78d8c9e9fcf334e]) successfully deleted
I20250115 14:52:02.296447 1751 tablet_replica.cc:331] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.296669 1751 raft_consensus.cc:2238] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.296847 1751 raft_consensus.cc:2267] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.297372 1751 ts_tablet_manager.cc:1905] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.297410 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4b828508be354eb4a8f42f5ba4b2e760"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.298501 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b828508be354eb4a8f42f5ba4b2e760 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.301695 1164 tablet_bootstrap.cc:492] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.304728 1164 tablet_bootstrap.cc:654] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.307261 1751 ts_tablet_manager.cc:1918] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.308651 1751 log.cc:1198] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b2c385edf4414157845b971aaf366831
I20250115 14:52:02.308988 1751 ts_tablet_manager.cc:1939] T b2c385edf4414157845b971aaf366831 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.255668 1844 raft_consensus.cc:491] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.273047 1843 raft_consensus.cc:491] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.277815 1838 raft_consensus.cc:491] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.309635 1843 raft_consensus.cc:513] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.309705 1838 raft_consensus.cc:513] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.310271 1838 leader_election.cc:290] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.310303 1843 leader_election.cc:290] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.310379 1838 raft_consensus.cc:491] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.310454 1838 raft_consensus.cc:513] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.310899 1838 leader_election.cc:290] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.311043 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7a487a42949bd9641c38d6c1fcfd3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.311120 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97c2f443ab040f9bc1924f45379165e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.311189 1844 raft_consensus.cc:513] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.311144 1955 raft_consensus.cc:491] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.311347 775 raft_consensus.cc:2463] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:02.311432 1955 raft_consensus.cc:513] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.311820 1844 leader_election.cc:290] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.312011 1955 leader_election.cc:290] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.312155 1751 tablet_replica.cc:331] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.312409 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b2c385edf4414157845b971aaf366831 (table default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]) successfully deleted
I20250115 14:52:02.309547 1876 raft_consensus.cc:491] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.312667 1876 raft_consensus.cc:513] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:02.312781 710 leader_election.cc:343] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 75788f8eb36b446e927197f624fbfdf5
I20250115 14:52:02.313213 1876 leader_election.cc:290] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:02.313911 848 leader_election.cc:343] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.314072 848 leader_election.cc:304] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:02.314905 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276abefcab6a422aa6467f00864c01f4" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.315258 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f07ef30cac2470d9b2f428338dd37a3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.315434 772 raft_consensus.cc:2463] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
W20250115 14:52:02.316084 848 leader_election.cc:343] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.316226 848 leader_election.cc:304] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:02.316550 1838 raft_consensus.cc:2798] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:02.316613 1838 raft_consensus.cc:491] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:02.316661 1838 raft_consensus.cc:3054] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.319109 1838 raft_consensus.cc:513] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.320106 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d97c2f443ab040f9bc1924f45379165e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:02.320282 772 raft_consensus.cc:3054] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.320389 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8fd4b26294e8ca6745b95f2e03231" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:02.320917 983 leader_election.cc:343] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.323330 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8fd4b26294e8ca6745b95f2e03231" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.284942 884 tablet_service.cc:1514] Processing DeleteTablet for tablet e6978223c1984a38bec5f2d06a9cf457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.288093 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:02.323891 984 leader_election.cc:343] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.324002 984 leader_election.cc:304] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.324215 1876 raft_consensus.cc:2743] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.324069 772 raft_consensus.cc:2463] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:02.324466 1844 raft_consensus.cc:2798] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:02.324537 1844 raft_consensus.cc:491] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:02.324597 1844 raft_consensus.cc:3054] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.325222 848 leader_election.cc:304] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:02.325541 1843 raft_consensus.cc:2798] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:02.325635 1843 raft_consensus.cc:695] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:02.325847 1843 consensus_queue.cc:237] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.327109 1838 leader_election.cc:290] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.336200 1751 raft_consensus.cc:2238] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.336549 1751 pending_rounds.cc:62] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 1 pending ops.
I20250115 14:52:02.336593 1751 pending_rounds.cc:69] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558346460123136 op_type: NO_OP noop_request { }
I20250115 14:52:02.336699 1751 raft_consensus.cc:2883] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:02.336771 1751 raft_consensus.cc:2267] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.344657 1981 tablet.cc:1721] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:02.345427 1982 tablet.cc:1721] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:02.349010 1751 ts_tablet_manager.cc:1905] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.354341 1162 tablet_bootstrap.cc:492] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.354527 1162 ts_tablet_manager.cc:1397] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.278s user 0.043s sys 0.061s
I20250115 14:52:02.354568 1751 ts_tablet_manager.cc:1918] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:02.354939 1751 log.cc:1198] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f8c81f47cd6e4a308cd62124db38d42f
I20250115 14:52:02.355329 1751 ts_tablet_manager.cc:1939] T f8c81f47cd6e4a308cd62124db38d42f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.355271 1162 raft_consensus.cc:357] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.355504 1162 raft_consensus.cc:383] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.355589 1162 raft_consensus.cc:738] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.355938 1162 consensus_queue.cc:260] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.356348 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f8c81f47cd6e4a308cd62124db38d42f (table default.Table_d8d3e6a881ce475a8f1319238ab8c610 [id=b6de0dbbced44c83af5f98a9d1020cfc]) successfully deleted
I20250115 14:52:02.357270 1162 ts_tablet_manager.cc:1428] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:02.357388 1164 tablet_bootstrap.cc:492] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.357630 1164 ts_tablet_manager.cc:1397] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.005s
I20250115 14:52:02.358736 1164 raft_consensus.cc:357] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.359018 1164 raft_consensus.cc:383] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.359151 1164 raft_consensus.cc:738] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.359481 1164 consensus_queue.cc:260] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.360503 1164 ts_tablet_manager.cc:1428] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:02.360801 1751 tablet_replica.cc:331] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.361027 1751 raft_consensus.cc:2238] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:02.361181 1164 tablet_bootstrap.cc:492] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.361320 1162 tablet_bootstrap.cc:492] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.361449 1751 raft_consensus.cc:2267] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.361951 1751 ts_tablet_manager.cc:1905] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.362840 1162 tablet_bootstrap.cc:654] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.365806 1983 tablet.cc:1721] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:02.372073 1751 ts_tablet_manager.cc:1918] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:02.372257 1164 tablet_bootstrap.cc:654] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.372414 1751 log.cc:1198] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/fe696560ba6b4a76ad013d9a3486e398
I20250115 14:52:02.374642 1164 tablet_bootstrap.cc:492] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.373267 1162 tablet_bootstrap.cc:492] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.374796 1164 ts_tablet_manager.cc:1397] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20250115 14:52:02.374039 1728 raft_consensus.cc:491] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.374938 1728 raft_consensus.cc:513] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.375502 1728 leader_election.cc:290] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.375593 1164 raft_consensus.cc:357] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.375861 1164 raft_consensus.cc:383] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.375924 1164 raft_consensus.cc:738] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.376003 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff155a5c2b0479a90e74993de6f402d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.376135 1162 ts_tablet_manager.cc:1397] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20250115 14:52:02.376152 1164 consensus_queue.cc:260] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:02.376605 983 leader_election.cc:343] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.376726 1164 ts_tablet_manager.cc:1428] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:02.376865 1162 raft_consensus.cc:357] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.377007 1164 tablet_bootstrap.cc:492] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.377111 1162 raft_consensus.cc:383] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.377215 1162 raft_consensus.cc:738] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.377328 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff155a5c2b0479a90e74993de6f402d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.377449 1162 consensus_queue.cc:260] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.378075 1162 ts_tablet_manager.cc:1428] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:02.378353 1162 tablet_bootstrap.cc:492] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.378551 1164 tablet_bootstrap.cc:654] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:02.380007 984 leader_election.cc:343] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.380193 984 leader_election.cc:304] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.380471 1728 raft_consensus.cc:2743] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.380694 1728 raft_consensus.cc:491] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.380885 1162 tablet_bootstrap.cc:654] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.380805 1728 raft_consensus.cc:513] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.381331 1728 leader_election.cc:290] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.381698 1164 tablet_bootstrap.cc:492] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.381819 1164 ts_tablet_manager.cc:1397] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250115 14:52:02.381944 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffa997ca6c46559830b1dedf2d1a66" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:02.382442 983 leader_election.cc:343] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.382499 1164 raft_consensus.cc:357] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.382692 1164 raft_consensus.cc:383] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.382766 1164 raft_consensus.cc:738] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.375054 1751 ts_tablet_manager.cc:1939] T fe696560ba6b4a76ad013d9a3486e398 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.382975 1164 consensus_queue.cc:260] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.383754 1164 ts_tablet_manager.cc:1428] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:02.383939 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet fe696560ba6b4a76ad013d9a3486e398 (table default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]) successfully deleted
I20250115 14:52:02.384066 1164 tablet_bootstrap.cc:492] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:02.384665 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a48227ec5aa846c3ac63bd29ed38be25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48227ec5aa846c3ac63bd29ed38be25 already in progress: creating tablet
I20250115 14:52:02.385254 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffa997ca6c46559830b1dedf2d1a66" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.385633 1164 tablet_bootstrap.cc:654] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:02.385684 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 95718d39a7824517a36c49c3b86fef30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95718d39a7824517a36c49c3b86fef30 already in progress: creating tablet
W20250115 14:52:02.386332 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 already in progress: creating tablet
I20250115 14:52:02.386641 1751 tablet_replica.cc:331] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.386919 1751 raft_consensus.cc:2238] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.387097 1751 raft_consensus.cc:2267] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.387646 1751 ts_tablet_manager.cc:1905] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:02.389179 984 leader_election.cc:343] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.389333 984 leader_election.cc:304] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.389593 1728 raft_consensus.cc:2743] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.389643 1162 tablet_bootstrap.cc:492] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.389737 1162 ts_tablet_manager.cc:1397] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20250115 14:52:02.390447 1162 raft_consensus.cc:357] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.390662 1162 raft_consensus.cc:383] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.390748 1162 raft_consensus.cc:738] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.390975 1162 consensus_queue.cc:260] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.391805 1162 ts_tablet_manager.cc:1428] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:02.392428 1162 tablet_bootstrap.cc:492] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.393432 1164 tablet_bootstrap.cc:492] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.393589 1164 ts_tablet_manager.cc:1397] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20250115 14:52:02.393633 1751 ts_tablet_manager.cc:1918] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.393797 1751 log.cc:1198] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/bc8337d181d44ac7bb82551a677b6831
I20250115 14:52:02.394133 1751 ts_tablet_manager.cc:1939] T bc8337d181d44ac7bb82551a677b6831 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.394378 1164 raft_consensus.cc:357] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.394577 1164 raft_consensus.cc:383] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.394654 1164 raft_consensus.cc:738] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.394874 1164 consensus_queue.cc:260] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.395319 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet bc8337d181d44ac7bb82551a677b6831 (table default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]) successfully deleted
I20250115 14:52:02.395524 1162 tablet_bootstrap.cc:654] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.395552 1164 ts_tablet_manager.cc:1428] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:02.395808 1164 tablet_bootstrap.cc:492] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.397325 1164 tablet_bootstrap.cc:654] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.401842 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7a487a42949bd9641c38d6c1fcfd3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:02.402029 1048 raft_consensus.cc:2463] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:02.402657 1162 tablet_bootstrap.cc:492] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.402776 1162 ts_tablet_manager.cc:1397] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250115 14:52:02.403199 847 leader_election.cc:304] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.403411 1162 raft_consensus.cc:357] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.403571 1162 raft_consensus.cc:383] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.403615 1162 raft_consensus.cc:738] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.403772 1162 consensus_queue.cc:260] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.403896 1843 raft_consensus.cc:2798] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:02.403973 1843 raft_consensus.cc:491] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:02.404036 1843 raft_consensus.cc:3054] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.404189 1162 ts_tablet_manager.cc:1428] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:02.396422 1751 tablet_replica.cc:331] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.404428 1751 raft_consensus.cc:2238] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.404577 1751 raft_consensus.cc:2267] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.405261 1751 ts_tablet_manager.cc:1905] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.406633 1843 raft_consensus.cc:513] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.409530 775 raft_consensus.cc:3054] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.411584 1751 ts_tablet_manager.cc:1918] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.411684 1751 log.cc:1198] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/59d8fd4b26294e8ca6745b95f2e03231
I20250115 14:52:02.411952 1751 ts_tablet_manager.cc:1939] T 59d8fd4b26294e8ca6745b95f2e03231 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.412921 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 59d8fd4b26294e8ca6745b95f2e03231 (table default.Table_98cb517939064897baf89d74e6e0355e [id=757cda13648a4e3d9b02d70555966106]) successfully deleted
I20250115 14:52:02.412969 1843 leader_election.cc:290] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.404423 1162 tablet_bootstrap.cc:492] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.414175 1843 consensus_queue.cc:1035] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:02.415851 1162 tablet_bootstrap.cc:654] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.415997 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7a487a42949bd9641c38d6c1fcfd3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:02.416541 848 leader_election.cc:343] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:02.417138 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a already in progress: creating tablet
W20250115 14:52:02.417794 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e6978223c1984a38bec5f2d06a9cf457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6978223c1984a38bec5f2d06a9cf457 already in progress: creating tablet
I20250115 14:52:02.418898 1164 tablet_bootstrap.cc:492] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.419049 1164 ts_tablet_manager.cc:1397] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s
I20250115 14:52:02.419867 1164 raft_consensus.cc:357] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.420069 1164 raft_consensus.cc:383] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.420166 1164 raft_consensus.cc:738] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.420426 1164 consensus_queue.cc:260] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.421030 1164 ts_tablet_manager.cc:1428] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:02.421289 1164 tablet_bootstrap.cc:492] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.421515 1728 raft_consensus.cc:491] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.421633 1728 raft_consensus.cc:513] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.422165 1728 leader_election.cc:290] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.422580 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caed629878594954842dfc6a40cc8b6d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.422922 1164 tablet_bootstrap.cc:654] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:02.423195 983 leader_election.cc:343] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.424531 1162 tablet_bootstrap.cc:492] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.424665 1162 ts_tablet_manager.cc:1397] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250115 14:52:02.425367 1162 raft_consensus.cc:357] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.425568 1162 raft_consensus.cc:383] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.425626 1162 raft_consensus.cc:738] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.425828 1162 consensus_queue.cc:260] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.426326 1162 ts_tablet_manager.cc:1428] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:02.426576 1162 tablet_bootstrap.cc:492] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.428169 1162 tablet_bootstrap.cc:654] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:02.429255 847 leader_election.cc:343] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 276abefcab6a422aa6467f00864c01f4
I20250115 14:52:02.429432 847 leader_election.cc:304] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.429705 1843 raft_consensus.cc:2743] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:02.430328 847 leader_election.cc:343] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6f07ef30cac2470d9b2f428338dd37a3
W20250115 14:52:02.430642 847 leader_election.cc:343] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d97c2f443ab040f9bc1924f45379165e
W20250115 14:52:02.430989 847 leader_election.cc:343] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d97c2f443ab040f9bc1924f45379165e
I20250115 14:52:02.431502 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e932d5b820f94130967eccce9e1542fa" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.431700 908 raft_consensus.cc:2463] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:02.432160 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75788f8eb36b446e927197f624fbfdf5" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.432919 1162 tablet_bootstrap.cc:492] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.433080 1162 ts_tablet_manager.cc:1397] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20250115 14:52:02.433112 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3a7a487a42949bd9641c38d6c1fcfd3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:02.433300 1048 raft_consensus.cc:3054] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.433885 1162 raft_consensus.cc:357] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.434062 1162 raft_consensus.cc:383] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.434108 1162 raft_consensus.cc:738] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.434299 1162 consensus_queue.cc:260] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.434757 1162 ts_tablet_manager.cc:1428] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:02.434998 1162 tablet_bootstrap.cc:492] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.435757 1048 raft_consensus.cc:2463] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:02.436800 1162 tablet_bootstrap.cc:654] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.440398 1981 tablet.cc:1721] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:02.450817 1983 tablet.cc:1721] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:02.452384 710 leader_election.cc:304] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.453902 1168 tablet_bootstrap.cc:492] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:02.454082 1168 ts_tablet_manager.cc:1397] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.229s user 0.004s sys 0.000s
W20250115 14:52:02.454773 710 leader_election.cc:343] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: Tablet not RUNNING: STOPPED
I20250115 14:52:02.454927 710 leader_election.cc:304] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.455173 1750 raft_consensus.cc:2743] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.455263 1955 raft_consensus.cc:2798] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:02.455933 1168 raft_consensus.cc:357] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.456135 1168 raft_consensus.cc:383] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.456179 1168 raft_consensus.cc:738] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.456378 1168 consensus_queue.cc:260] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.457247 1168 ts_tablet_manager.cc:1428] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:02.457474 1955 raft_consensus.cc:491] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:02.457612 1955 raft_consensus.cc:3054] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:02.460886 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:40227, user_credentials={real_user=slave}} blocked reactor thread for 190243us
I20250115 14:52:02.464344 1982 tablet.cc:1721] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:02.465729 1167 ts_tablet_manager.cc:1918] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.465901 1167 log.cc:1198] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/75788f8eb36b446e927197f624fbfdf5
I20250115 14:52:02.466256 1167 ts_tablet_manager.cc:1939] T 75788f8eb36b446e927197f624fbfdf5 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:02.466497 743 tablet_service.cc:1514] Processing DeleteTablet for tablet e6978223c1984a38bec5f2d06a9cf457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.467036 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 462) took 2434 ms. Trace:
I20250115 14:52:02.467123 1167 rpcz_store.cc:276] 0115 14:52:00.032467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.033097 (+ 630us) service_pool.cc:225] Handling call
0115 14:52:02.127667 (+2094570us) ts_tablet_manager.cc:1145] Deleting tablet 75788f8eb36b446e927197f624fbfdf5
0115 14:52:02.127675 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.352452 (+224777us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.465603 (+113151us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.467014 (+ 1411us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2094358}
I20250115 14:52:02.467626 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 75788f8eb36b446e927197f624fbfdf5 (table default.Table_875d7bd9588344008279ca89c1ad52be [id=7f3b54e6dae14efcb524f4f03c0a57b1]) successfully deleted
W20250115 14:52:02.472482 847 consensus_peers.cc:487] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8c81f47cd6e4a308cd62124db38d42f. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:02.472671 847 leader_election.cc:304] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.472915 1843 raft_consensus.cc:2798] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:02.473008 1843 raft_consensus.cc:695] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:02.473232 1843 consensus_queue.cc:237] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.475561 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aad920dd9744d9b87a8234c92232d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.475801 1166 tablet_replica.cc:331] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.486728 1955 raft_consensus.cc:513] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.487360 1955 leader_election.cc:290] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.487535 749 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4102b1f83e414d93a6a68315e36d16 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.487740 739 tablet_service.cc:1514] Processing DeleteTablet for tablet e932d5b820f94130967eccce9e1542fa with delete_type TABLET_DATA_DELETED (replaced by tablet fc3d2ca7cd7c48a79ed1396fbbdc6866) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.487834 1168 tablet_bootstrap.cc:492] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:02.487876 745 tablet_service.cc:1514] Processing DeleteTablet for tablet f847f92cd122423aa93f1afa701b0bbb with delete_type TABLET_DATA_DELETED (replaced by tablet 83415f1c07ad495cabd287bdcd54dad4) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.488008 1750 raft_consensus.cc:491] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.488117 1750 raft_consensus.cc:513] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.488627 1750 leader_election.cc:290] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.488765 1750 raft_consensus.cc:491] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.488838 1166 raft_consensus.cc:2238] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.488893 1750 raft_consensus.cc:513] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.488996 1166 raft_consensus.cc:2267] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.489377 1750 leader_election.cc:290] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.489539 1166 ts_tablet_manager.cc:1905] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:02.491329 710 leader_election.cc:343] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e932d5b820f94130967eccce9e1542fa
W20250115 14:52:02.491518 710 leader_election.cc:343] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 9145bbaf3e234f079e355454f3ce28cb
W20250115 14:52:02.492100 710 leader_election.cc:343] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 0de6f6f014074c3d8775cb5e4121ccb0
I20250115 14:52:02.492353 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.492538 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caed629878594954842dfc6a40cc8b6d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.493387 984 leader_election.cc:343] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.493454 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9f327ffa731c4c949b24a34dc54ae260 (table default.Table_461138269c764aceb6cd3baa42a84638 [id=7111d7db2d40429db3c4e77c694995a2]) successfully deleted
I20250115 14:52:02.493552 984 leader_election.cc:304] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.493808 1876 raft_consensus.cc:2743] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.494906 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e932d5b820f94130967eccce9e1542fa" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:02.495297 913 raft_consensus.cc:3054] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.495815 1166 ts_tablet_manager.cc:1918] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.495976 1166 log.cc:1198] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/79aad920dd9744d9b87a8234c92232d8
I20250115 14:52:02.496196 1876 raft_consensus.cc:491] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.496327 1166 ts_tablet_manager.cc:1939] T 79aad920dd9744d9b87a8234c92232d8 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:02.496336 1876 raft_consensus.cc:513] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.496912 1876 leader_election.cc:290] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.497149 1166 tablet_replica.cc:331] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.497246 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de6f6f014074c3d8775cb5e4121ccb0" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.497332 1166 raft_consensus.cc:2238] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.497489 1166 raft_consensus.cc:2267] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.497581 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb19d39258e444799d403a43475d9c8a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:02.497995 983 leader_election.cc:343] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.498057 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9145bbaf3e234f079e355454f3ce28cb" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.498199 1166 ts_tablet_manager.cc:1905] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.498476 710 leader_election.cc:304] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.498858 1750 raft_consensus.cc:491] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.498988 1750 raft_consensus.cc:513] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.499992 1750 leader_election.cc:290] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.500128 1750 raft_consensus.cc:2743] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.500945 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb19d39258e444799d403a43475d9c8a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.501123 710 leader_election.cc:343] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 1f1df9dc743342b4b7bdb1a8d3fcd5b0
W20250115 14:52:02.501433 984 leader_election.cc:343] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.501581 984 leader_election.cc:304] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.501837 1876 raft_consensus.cc:2743] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.502133 913 raft_consensus.cc:2463] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:02.502465 710 leader_election.cc:304] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.502899 1955 raft_consensus.cc:2743] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.503314 710 leader_election.cc:304] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.503710 1955 raft_consensus.cc:2798] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:02.503841 1955 raft_consensus.cc:695] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:02.504069 1955 consensus_queue.cc:237] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.506140 1876 raft_consensus.cc:491] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.506268 1876 raft_consensus.cc:513] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.506845 1876 leader_election.cc:290] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:02.509111 710 leader_election.cc:343] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 1f1df9dc743342b4b7bdb1a8d3fcd5b0
I20250115 14:52:02.509245 710 leader_election.cc:304] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.509642 1750 raft_consensus.cc:2743] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.510879 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ad2b7350294375bd30fb4b46906491" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.510262 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ad2b7350294375bd30fb4b46906491" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.511315 983 leader_election.cc:343] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:02.511641 984 leader_election.cc:343] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.511749 984 leader_election.cc:304] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.511956 1876 raft_consensus.cc:2743] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:02.512274 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e6978223c1984a38bec5f2d06a9cf457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6978223c1984a38bec5f2d06a9cf457 already in progress: creating tablet
I20250115 14:52:02.512634 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 79aad920dd9744d9b87a8234c92232d8 (table default.Table_4e0b412e80014674bc22990db5953cc6 [id=66d6a31a1f8546a1bd7e269ce97ec9f7]) successfully deleted
I20250115 14:52:02.504259 1166 ts_tablet_manager.cc:1918] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.516266 1166 log.cc:1198] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/bd4102b1f83e414d93a6a68315e36d16
I20250115 14:52:02.516666 1166 ts_tablet_manager.cc:1939] T bd4102b1f83e414d93a6a68315e36d16 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:02.517506 1166 tablet_replica.cc:331] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.535635 1168 tablet_bootstrap.cc:654] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.536576 1166 raft_consensus.cc:2238] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:02.537288 1166 pending_rounds.cc:62] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:02.537387 1166 pending_rounds.cc:69] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558351380549632 op_type: NO_OP noop_request { }
I20250115 14:52:02.537523 1166 raft_consensus.cc:2883] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:02.537663 1166 raft_consensus.cc:2267] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:02.537459 2003 consensus_queue.cc:1159] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d2d7ecece2ab49e1be7079216517d13b" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250115 14:52:02.538082 1167 tablet_replica.cc:331] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:02.538342 1167 raft_consensus.cc:2238] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:02.538821 1167 pending_rounds.cc:62] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:02.538918 1167 pending_rounds.cc:69] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558346164625408 op_type: NO_OP noop_request { }
I20250115 14:52:02.539064 1167 raft_consensus.cc:2883] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:02.539494 1167 raft_consensus.cc:2267] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.540145 1167 ts_tablet_manager.cc:1905] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.545198 1166 ts_tablet_manager.cc:1905] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.551877 1166 ts_tablet_manager.cc:1918] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:02.552017 1166 log.cc:1198] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e932d5b820f94130967eccce9e1542fa
I20250115 14:52:02.554344 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet bd4102b1f83e414d93a6a68315e36d16 (table default.Table_d0d86551c8474fb98ad74cb89ff84e28 [id=becd439cb45d48c3974b9ae72768542a]) successfully deleted
I20250115 14:52:02.560602 1166 ts_tablet_manager.cc:1939] T e932d5b820f94130967eccce9e1542fa P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:52:02.564642 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 consensus state has no leader
I20250115 14:52:02.564947 446 master_service.cc:946] ReplaceTablet: received request to replace tablet e244689cef6844e2935becd265eecad0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:02.565161 1166 tablet_replica.cc:331] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.565402 1166 raft_consensus.cc:2238] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:02.565646 1166 pending_rounds.cc:62] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:02.565725 1166 pending_rounds.cc:69] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558349613129728 op_type: NO_OP noop_request { }
I20250115 14:52:02.565851 1166 raft_consensus.cc:2883] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:02.565963 1166 raft_consensus.cc:2267] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.566772 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e932d5b820f94130967eccce9e1542fa (table default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]) successfully deleted
I20250115 14:52:02.566836 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7d612b424bea408cb6691c7c200cf3a6
I20250115 14:52:02.567916 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d612b424bea408cb6691c7c200cf3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.567996 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d612b424bea408cb6691c7c200cf3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.568531 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d612b424bea408cb6691c7c200cf3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:02.568830 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7d612b424bea408cb6691c7c200cf3a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d612b424bea408cb6691c7c200cf3a6 already in progress: creating tablet
I20250115 14:52:02.570130 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 07a000ab73dd46ac8e134784a156040a from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:02.570782 1168 tablet_bootstrap.cc:492] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:02.570915 1168 ts_tablet_manager.cc:1397] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.000s
I20250115 14:52:02.570888 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9c676839fdd84942afad4ab037b135ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.571455 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c676839fdd84942afad4ab037b135ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.571614 1168 raft_consensus.cc:357] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.571781 1168 raft_consensus.cc:383] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.571826 1168 raft_consensus.cc:738] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.572058 1168 consensus_queue.cc:260] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.572641 1168 ts_tablet_manager.cc:1428] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:02.573005 1168 tablet_bootstrap.cc:492] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:02.574296 1168 tablet_bootstrap.cc:654] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.576797 1164 tablet_bootstrap.cc:492] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.576946 1164 ts_tablet_manager.cc:1397] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.156s user 0.030s sys 0.042s
I20250115 14:52:02.577239 454 master_service.cc:946] ReplaceTablet: received request to replace tablet c574dba6b2a04915bdd871de4dff4df0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:02.577592 1164 raft_consensus.cc:357] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.577797 1164 raft_consensus.cc:383] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.577857 1164 raft_consensus.cc:738] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.578092 1164 consensus_queue.cc:260] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.578410 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 2560d42c15c64b4c9c0cfe95706b4d9e from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:02.578586 1164 ts_tablet_manager.cc:1428] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:02.578995 1166 ts_tablet_manager.cc:1905] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.579435 1844 raft_consensus.cc:513] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.580768 1164 tablet_bootstrap.cc:492] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.581663 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f07ef30cac2470d9b2f428338dd37a3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:02.581867 776 raft_consensus.cc:3054] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.584617 776 raft_consensus.cc:2463] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:02.585258 848 leader_election.cc:304] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:02.585513 1843 raft_consensus.cc:2798] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:02.585608 1843 raft_consensus.cc:695] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:02.585819 1843 consensus_queue.cc:237] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.586328 1164 tablet_bootstrap.cc:654] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:02.587325 847 leader_election.cc:343] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6f07ef30cac2470d9b2f428338dd37a3
I20250115 14:52:02.587545 452 master_service.cc:946] ReplaceTablet: received request to replace tablet b76d1bbf90c6416ca60805c9d8053bd4 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:02.589012 1162 tablet_bootstrap.cc:492] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.589151 1162 ts_tablet_manager.cc:1397] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.154s user 0.023s sys 0.035s
I20250115 14:52:02.589844 1162 raft_consensus.cc:357] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.590029 1162 raft_consensus.cc:383] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.590077 1162 raft_consensus.cc:738] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.590256 1162 consensus_queue.cc:260] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.590692 1162 ts_tablet_manager.cc:1428] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:02.590942 1162 tablet_bootstrap.cc:492] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.592435 1162 tablet_bootstrap.cc:654] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.593119 1975 raft_consensus.cc:491] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.593242 1975 raft_consensus.cc:513] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.593788 1975 leader_election.cc:290] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.594177 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7233ebf3bae1462d8eeb3669b39944c0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:02.594399 1049 raft_consensus.cc:2463] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:02.594949 847 leader_election.cc:304] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:02.595317 1975 raft_consensus.cc:2798] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:02.595386 1975 raft_consensus.cc:491] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:02.595439 1975 raft_consensus.cc:3054] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.597882 1975 raft_consensus.cc:513] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.598779 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7233ebf3bae1462d8eeb3669b39944c0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:02.598989 775 raft_consensus.cc:3054] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.601111 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7233ebf3bae1462d8eeb3669b39944c0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:02.601261 1049 raft_consensus.cc:3054] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.601788 1166 ts_tablet_manager.cc:1918] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:02.601903 1166 log.cc:1198] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f847f92cd122423aa93f1afa701b0bbb
I20250115 14:52:02.603621 1049 raft_consensus.cc:2463] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:02.605120 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
I20250115 14:52:02.605114 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 0d499d1b14054037a57fab482a171eca (DEFAULT_TABLE table=default.Table_dc6355486b8a4565a2749cf74bec9a2b [id=f085399ab0e743a8aaf1417f20c571f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.605340 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_7b4753acbcca4cdc94831798b3d47baa [id=7628dbb6818f443680a97fef665d6d2b]). Expected version 2 got 0
I20250115 14:52:02.605463 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d499d1b14054037a57fab482a171eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.605490 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]). Expected version 1 got 0
I20250115 14:52:02.605660 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet dff155a5c2b0479a90e74993de6f402d (table default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]). Expected version 1 got 0
I20250115 14:52:02.607411 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_345eac8bafcc47b59e881aa44718957e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.607882 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_345eac8bafcc47b59e881aa44718957e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.608331 847 leader_election.cc:304] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:02.609177 1838 raft_consensus.cc:2798] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:02.609669 879 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7a5cbd9a78485a816d7fecdc23106a (DEFAULT_TABLE table=default.Table_1f3c5d983c50484da1dc4417e5698154 [id=290d60ba19284a159fd6d30702ba32fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.609843 884 tablet_service.cc:1467] Processing CreateTablet for tablet db5518ac647c472bb47ed741022c820a (DEFAULT_TABLE table=default.Table_0b1e4ccdc6ce444195b8bf86c86ff87b [id=9b5f0afc3d0e404081400d7c8eaa5389]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.610033 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7a5cbd9a78485a816d7fecdc23106a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.610411 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db5518ac647c472bb47ed741022c820a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.612520 878 tablet_service.cc:1467] Processing CreateTablet for tablet 71465ac2f92f4737803674a4249c1f85 (DEFAULT_TABLE table=default.Table_4d817129c1cf46a6bde83e97a293603b [id=8256838d8e8246ac8ed886263eecfc5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.612809 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71465ac2f92f4737803674a4249c1f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.615989 1975 leader_election.cc:290] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.616372 1166 ts_tablet_manager.cc:1939] T f847f92cd122423aa93f1afa701b0bbb P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:02.616493 889 tablet_service.cc:1467] Processing CreateTablet for tablet 4934bdde6023419fafd750920bcfe3d4 (DEFAULT_TABLE table=default.Table_afd9df7559aa485789c8e7bd5d43c3ef [id=15a1f705007143a4a8230e87e3042265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.616575 1955 raft_consensus.cc:491] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.616789 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4934bdde6023419fafd750920bcfe3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.616699 1955 raft_consensus.cc:513] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.617338 1955 leader_election.cc:290] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.619429 775 raft_consensus.cc:2463] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:02.620507 882 tablet_service.cc:1467] Processing CreateTablet for tablet 856533ed254a49b4a461ed1ad3eb405e (DEFAULT_TABLE table=default.Table_12287ed111bb450dbe84bdb433fc746e [id=ac030a4c81a444b399fedcdc1a76a47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.620798 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856533ed254a49b4a461ed1ad3eb405e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.620815 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7233ebf3bae1462d8eeb3669b39944c0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.621045 776 raft_consensus.cc:2371] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1. Re-sending same reply.
I20250115 14:52:02.624509 891 tablet_service.cc:1467] Processing CreateTablet for tablet 69c0368e67624a6e8d1f6a4a6275466b (DEFAULT_TABLE table=default.Table_607f5ab1bfce4111821d459c200808f7 [id=6504ea1fc6df4f97afd78c878aa2adca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.624814 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c0368e67624a6e8d1f6a4a6275466b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.628329 1844 leader_election.cc:290] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.628523 887 tablet_service.cc:1467] Processing CreateTablet for tablet 6107d16e09b1477d84f089a92cb0f813 (DEFAULT_TABLE table=default.Table_e82073df06d449c3870cf6d7b8133825 [id=7cfa67de4c36418e93bc35df5ac73425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.628887 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6107d16e09b1477d84f089a92cb0f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.609684 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0d499d1b14054037a57fab482a171eca (DEFAULT_TABLE table=default.Table_dc6355486b8a4565a2749cf74bec9a2b [id=f085399ab0e743a8aaf1417f20c571f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.633138 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d499d1b14054037a57fab482a171eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.637514 1019 tablet_service.cc:1467] Processing CreateTablet for tablet db5518ac647c472bb47ed741022c820a (DEFAULT_TABLE table=default.Table_0b1e4ccdc6ce444195b8bf86c86ff87b [id=9b5f0afc3d0e404081400d7c8eaa5389]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.637861 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db5518ac647c472bb47ed741022c820a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.639223 1162 tablet_bootstrap.cc:492] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.639341 1162 ts_tablet_manager.cc:1397] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.014s
I20250115 14:52:02.640012 1162 raft_consensus.cc:357] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.640177 1162 raft_consensus.cc:383] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.640219 1162 raft_consensus.cc:738] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.640461 1162 consensus_queue.cc:260] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.640568 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7a5cbd9a78485a816d7fecdc23106a (DEFAULT_TABLE table=default.Table_1f3c5d983c50484da1dc4417e5698154 [id=290d60ba19284a159fd6d30702ba32fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.640868 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7a5cbd9a78485a816d7fecdc23106a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.641337 1162 ts_tablet_manager.cc:1428] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:02.641556 1162 tablet_bootstrap.cc:492] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.643059 1162 tablet_bootstrap.cc:654] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.644511 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 71465ac2f92f4737803674a4249c1f85 (DEFAULT_TABLE table=default.Table_4d817129c1cf46a6bde83e97a293603b [id=8256838d8e8246ac8ed886263eecfc5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.644800 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71465ac2f92f4737803674a4249c1f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.647565 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d8fd4b26294e8ca6745b95f2e03231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.647882 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 856533ed254a49b4a461ed1ad3eb405e (DEFAULT_TABLE table=default.Table_12287ed111bb450dbe84bdb433fc746e [id=ac030a4c81a444b399fedcdc1a76a47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.648151 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856533ed254a49b4a461ed1ad3eb405e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.649432 1162 tablet_bootstrap.cc:492] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.649528 1162 ts_tablet_manager.cc:1397] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20250115 14:52:02.650166 1162 raft_consensus.cc:357] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.650317 1162 raft_consensus.cc:383] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.650357 1162 raft_consensus.cc:738] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.651850 1162 consensus_queue.cc:260] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.652386 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet d195be2d543f4a8693045a024912c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.652722 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 69c0368e67624a6e8d1f6a4a6275466b (DEFAULT_TABLE table=default.Table_607f5ab1bfce4111821d459c200808f7 [id=6504ea1fc6df4f97afd78c878aa2adca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.652864 1162 ts_tablet_manager.cc:1428] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:02.652994 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c0368e67624a6e8d1f6a4a6275466b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.653124 1162 tablet_bootstrap.cc:492] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.654650 1162 tablet_bootstrap.cc:654] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.656522 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 6107d16e09b1477d84f089a92cb0f813 (DEFAULT_TABLE table=default.Table_e82073df06d449c3870cf6d7b8133825 [id=7cfa67de4c36418e93bc35df5ac73425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.656826 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6107d16e09b1477d84f089a92cb0f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.660694 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 4934bdde6023419fafd750920bcfe3d4 (DEFAULT_TABLE table=default.Table_afd9df7559aa485789c8e7bd5d43c3ef [id=15a1f705007143a4a8230e87e3042265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.661007 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4934bdde6023419fafd750920bcfe3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.662024 1162 tablet_bootstrap.cc:492] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.663045 1162 ts_tablet_manager.cc:1397] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20250115 14:52:02.664677 1751 tablet_replica.cc:331] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.664857 1751 raft_consensus.cc:2238] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.664991 1751 raft_consensus.cc:2267] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.665539 1162 raft_consensus.cc:357] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.666893 1162 raft_consensus.cc:383] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.666968 1162 raft_consensus.cc:738] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.667083 1751 ts_tablet_manager.cc:1905] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.667496 1162 consensus_queue.cc:260] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.668949 1162 ts_tablet_manager.cc:1428] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:52:02.669229 1162 tablet_bootstrap.cc:492] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.670639 1162 tablet_bootstrap.cc:654] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.675591 1876 raft_consensus.cc:491] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:52:02.675700 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 59d8fd4b26294e8ca6745b95f2e03231 because the tablet was not found. No further retry: Not found: Tablet not found: 59d8fd4b26294e8ca6745b95f2e03231
I20250115 14:52:02.675750 1876 raft_consensus.cc:513] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.676342 1876 leader_election.cc:290] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.677706 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e94ea76507f4646909ac0c1b2c3136a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.677891 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]). Expected version 2 got 0
I20250115 14:52:02.678093 449 catalog_manager.cc:5526] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:02.678520 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a3a7a487a42949bd9641c38d6c1fcfd3 (table default.Table_8b9b25dd8a85434a83d2641ffbc5189e [id=738170c5571a4d958944ed432428e99d]). Expected version 3 got 0
W20250115 14:52:02.678575 983 leader_election.cc:343] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.678879 1975 raft_consensus.cc:491] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.678997 1975 raft_consensus.cc:513] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.679533 1975 leader_election.cc:290] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:02.679548 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e94ea76507f4646909ac0c1b2c3136a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.680241 984 leader_election.cc:343] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.680383 984 leader_election.cc:304] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.680526 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276abefcab6a422aa6467f00864c01f4" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.680703 913 raft_consensus.cc:2463] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:02.681154 710 leader_election.cc:304] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:02.681461 1955 raft_consensus.cc:2798] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:02.681402 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7100e6a51ed4e178e86138f10d5ffc2" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.681566 1955 raft_consensus.cc:491] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:02.681649 1955 raft_consensus.cc:3054] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:02.681881 848 leader_election.cc:343] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:02.682056 847 leader_election.cc:343] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e7100e6a51ed4e178e86138f10d5ffc2
I20250115 14:52:02.682123 847 leader_election.cc:304] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.682456 1844 raft_consensus.cc:2743] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.682647 1876 raft_consensus.cc:2743] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.687817 1162 tablet_bootstrap.cc:492] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.687958 1162 ts_tablet_manager.cc:1397] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
I20250115 14:52:02.688663 1162 raft_consensus.cc:357] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.688866 1162 raft_consensus.cc:383] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.684280 1955 raft_consensus.cc:513] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:02.695783 710 leader_election.cc:343] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 276abefcab6a422aa6467f00864c01f4
I20250115 14:52:02.696851 741 tablet_service.cc:1467] Processing CreateTablet for tablet 0d499d1b14054037a57fab482a171eca (DEFAULT_TABLE table=default.Table_dc6355486b8a4565a2749cf74bec9a2b [id=f085399ab0e743a8aaf1417f20c571f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.697260 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d499d1b14054037a57fab482a171eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.700827 1876 raft_consensus.cc:491] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.700951 1876 raft_consensus.cc:513] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.701506 1876 leader_election.cc:290] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.703634 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf94077e0e343b8938b081102dfd2af" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.704185 984 leader_election.cc:343] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.704684 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf94077e0e343b8938b081102dfd2af" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:02.705233 983 leader_election.cc:343] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.705329 983 leader_election.cc:304] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.705828 1876 raft_consensus.cc:2743] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:02.709019 710 leader_election.cc:343] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 276abefcab6a422aa6467f00864c01f4
I20250115 14:52:02.709583 1955 leader_election.cc:290] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.710103 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "276abefcab6a422aa6467f00864c01f4" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:02.710290 913 raft_consensus.cc:3054] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.712913 913 raft_consensus.cc:2463] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:02.710755 745 tablet_service.cc:1467] Processing CreateTablet for tablet db5518ac647c472bb47ed741022c820a (DEFAULT_TABLE table=default.Table_0b1e4ccdc6ce444195b8bf86c86ff87b [id=9b5f0afc3d0e404081400d7c8eaa5389]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.718932 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db5518ac647c472bb47ed741022c820a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.719775 1162 raft_consensus.cc:738] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.720005 1162 consensus_queue.cc:260] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.720686 1162 ts_tablet_manager.cc:1428] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.033s user 0.011s sys 0.004s
I20250115 14:52:02.720924 1162 tablet_bootstrap.cc:492] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.711117 752 tablet_service.cc:1467] Processing CreateTablet for tablet 4934bdde6023419fafd750920bcfe3d4 (DEFAULT_TABLE table=default.Table_afd9df7559aa485789c8e7bd5d43c3ef [id=15a1f705007143a4a8230e87e3042265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.711236 739 tablet_service.cc:1467] Processing CreateTablet for tablet 2f7a5cbd9a78485a816d7fecdc23106a (DEFAULT_TABLE table=default.Table_1f3c5d983c50484da1dc4417e5698154 [id=290d60ba19284a159fd6d30702ba32fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.722681 1162 tablet_bootstrap.cc:654] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.722856 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4934bdde6023419fafd750920bcfe3d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.725466 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f7a5cbd9a78485a816d7fecdc23106a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.711410 743 tablet_service.cc:1467] Processing CreateTablet for tablet 856533ed254a49b4a461ed1ad3eb405e (DEFAULT_TABLE table=default.Table_12287ed111bb450dbe84bdb433fc746e [id=ac030a4c81a444b399fedcdc1a76a47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.727380 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856533ed254a49b4a461ed1ad3eb405e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.730265 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f847f92cd122423aa93f1afa701b0bbb (table default.Table_458a573b4a4b4714950723b426f0d265 [id=1fd54f7dafdc4777bd06982391452889]) successfully deleted
I20250115 14:52:02.711561 749 tablet_service.cc:1467] Processing CreateTablet for tablet 71465ac2f92f4737803674a4249c1f85 (DEFAULT_TABLE table=default.Table_4d817129c1cf46a6bde83e97a293603b [id=8256838d8e8246ac8ed886263eecfc5b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:02.730509 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a already in progress: creating tablet
I20250115 14:52:02.730700 1162 tablet_bootstrap.cc:492] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
W20250115 14:52:02.730720 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7d612b424bea408cb6691c7c200cf3a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d612b424bea408cb6691c7c200cf3a6 already in progress: creating tablet
I20250115 14:52:02.730772 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71465ac2f92f4737803674a4249c1f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.730829 1162 ts_tablet_manager.cc:1397] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20250115 14:52:02.732108 1162 raft_consensus.cc:357] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.732348 1162 raft_consensus.cc:383] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.732395 1162 raft_consensus.cc:738] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.732600 1162 consensus_queue.cc:260] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.733531 1162 ts_tablet_manager.cc:1428] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:02.711673 751 tablet_service.cc:1467] Processing CreateTablet for tablet 69c0368e67624a6e8d1f6a4a6275466b (DEFAULT_TABLE table=default.Table_607f5ab1bfce4111821d459c200808f7 [id=6504ea1fc6df4f97afd78c878aa2adca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.734203 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69c0368e67624a6e8d1f6a4a6275466b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.734202 1162 tablet_bootstrap.cc:492] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.735803 1162 tablet_bootstrap.cc:654] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.713341 710 leader_election.cc:304] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.737252 1955 raft_consensus.cc:2798] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:02.737680 1955 raft_consensus.cc:695] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:02.737900 750 tablet_service.cc:1467] Processing CreateTablet for tablet 6107d16e09b1477d84f089a92cb0f813 (DEFAULT_TABLE table=default.Table_e82073df06d449c3870cf6d7b8133825 [id=7cfa67de4c36418e93bc35df5ac73425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.737975 1955 consensus_queue.cc:237] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.738286 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6107d16e09b1477d84f089a92cb0f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.742590 1751 ts_tablet_manager.cc:1918] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.756017 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_81803594c5c741619ae708256092f827 [id=caa6adb7563c41088e5120993c46c54f] alter complete (version 2)
I20250115 14:52:02.761541 1751 log.cc:1198] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d195be2d543f4a8693045a024912c43a
I20250115 14:52:02.761926 1751 ts_tablet_manager.cc:1939] T d195be2d543f4a8693045a024912c43a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.762969 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d195be2d543f4a8693045a024912c43a (table default.Table_bba588d115d34226951843abdb9b6ebe [id=e99d33d2241343e2818d4217babf9a49]) successfully deleted
I20250115 14:52:02.767765 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.768126 1166 tablet_replica.cc:331] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.767804 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9145bbaf3e234f079e355454f3ce28cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.768738 1166 raft_consensus.cc:2238] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.768916 1166 raft_consensus.cc:2267] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.769472 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f07ef30cac2470d9b2f428338dd37a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.769691 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 75788f8eb36b446e927197f624fbfdf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.770258 752 tablet_service.cc:1514] Processing DeleteTablet for tablet d97c2f443ab040f9bc1924f45379165e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.770431 751 tablet_service.cc:1514] Processing DeleteTablet for tablet d4c984613c7d45d5b7c513cd900c9558 with delete_type TABLET_DATA_DELETED (replaced by tablet 1218f0bdb1e14bd78d76c6a06805f38c) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.770701 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 276abefcab6a422aa6467f00864c01f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.771003 739 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c81f47cd6e4a308cd62124db38d42f with delete_type TABLET_DATA_DELETED (replaced by tablet 580742eb5dac4dba9995809890e0e315) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.771193 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de6f6f014074c3d8775cb5e4121ccb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.771701 1166 ts_tablet_manager.cc:1905] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.771859 1167 ts_tablet_manager.cc:1918] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:02.771976 1167 log.cc:1198] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/0de6f6f014074c3d8775cb5e4121ccb0
I20250115 14:52:02.772336 1167 ts_tablet_manager.cc:1939] T 0de6f6f014074c3d8775cb5e4121ccb0 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:02.773485 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9915ea9ed9b749c8957dd5fb5152b5e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.774052 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9915ea9ed9b749c8957dd5fb5152b5e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.775174 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5b85d63974ab418eb1921dd605fbf2b0
I20250115 14:52:02.775209 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 463) took 2725 ms. Trace:
I20250115 14:52:02.775274 1167 rpcz_store.cc:276] 0115 14:52:00.049887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.049939 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:02.538013 (+2488074us) ts_tablet_manager.cc:1145] Deleting tablet 0de6f6f014074c3d8775cb5e4121ccb0
0115 14:52:02.538028 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.543545 (+ 5517us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.771748 (+228203us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.775190 (+ 3442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2487843}
I20250115 14:52:02.775570 1167 tablet_replica.cc:331] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:02.775733 1167 raft_consensus.cc:2238] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.775869 1167 raft_consensus.cc:2267] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.775893 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_87cf8b44f8fb4fbda80f72656bfaa074" } new_table_name: "default.Table_bbd25ad975704fbcae339bf950fd1052" modify_external_catalogs: true
I20250115 14:52:02.776363 1167 ts_tablet_manager.cc:1905] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.777070 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6942d102069d4c33b23fa2c0b29f94a6
I20250115 14:52:02.777617 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2c94d0bb9dd54c5195b1fcbf7cec15fd" } new_table_name: "default.Table_57e02beccdd64ec6ad08a1f3d0908b65" modify_external_catalogs: true
I20250115 14:52:02.778934 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e847214582ef4c838ae73922b0c49ef9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.779393 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e847214582ef4c838ae73922b0c49ef9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.780462 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e244689cef6844e2935becd265eecad0
I20250115 14:52:02.780872 446 catalog_manager.cc:6596] ReplaceTablet: tablet e244689cef6844e2935becd265eecad0 deleted and replaced by tablet 2a1a69b6de5d433f82579b1259548fd2
I20250115 14:52:02.781191 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_169aa44347fe4b50b95e7c62b9604402"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.781683 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_169aa44347fe4b50b95e7c62b9604402 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.782740 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c574dba6b2a04915bdd871de4dff4df0
I20250115 14:52:02.783103 454 catalog_manager.cc:6596] ReplaceTablet: tablet c574dba6b2a04915bdd871de4dff4df0 deleted and replaced by tablet ca5cc55b50ee4953800f10efe0ffb546
I20250115 14:52:02.783347 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_116f78c6117c4cf4b8db61d75ae62013" } new_table_name: "default.Table_aacabd4362854aeaa394ec57852c2ccc" modify_external_catalogs: true
I20250115 14:52:02.784449 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b76d1bbf90c6416ca60805c9d8053bd4
I20250115 14:52:02.784833 452 catalog_manager.cc:6596] ReplaceTablet: tablet b76d1bbf90c6416ca60805c9d8053bd4 deleted and replaced by tablet 5e10fe1445be457a8c848c30771af866
I20250115 14:52:02.785586 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d548be616a2c4f01ac52eadd5c009eaf" } modify_external_catalogs: true
I20250115 14:52:02.785733 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d548be616a2c4f01ac52eadd5c009eaf" } modify_external_catalogs: true
I20250115 14:52:02.786403 1166 ts_tablet_manager.cc:1918] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.786545 1166 log.cc:1198] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/1f1df9dc743342b4b7bdb1a8d3fcd5b0
I20250115 14:52:02.786926 1166 ts_tablet_manager.cc:1939] T 1f1df9dc743342b4b7bdb1a8d3fcd5b0 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:02.787886 1166 tablet_replica.cc:331] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.788072 1166 raft_consensus.cc:2238] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.788415 776 raft_consensus.cc:3054] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.790288 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 0de6f6f014074c3d8775cb5e4121ccb0 (table default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]) successfully deleted
I20250115 14:52:02.790401 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet e244689cef6844e2935becd265eecad0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1a69b6de5d433f82579b1259548fd2) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.790601 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet c574dba6b2a04915bdd871de4dff4df0 with delete_type TABLET_DATA_DELETED (replaced by tablet ca5cc55b50ee4953800f10efe0ffb546) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.790803 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 6942d102069d4c33b23fa2c0b29f94a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.790995 883 tablet_service.cc:1514] Processing DeleteTablet for tablet c574dba6b2a04915bdd871de4dff4df0 with delete_type TABLET_DATA_DELETED (replaced by tablet ca5cc55b50ee4953800f10efe0ffb546) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.791174 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b85d63974ab418eb1921dd605fbf2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.791340 878 tablet_service.cc:1514] Processing DeleteTablet for tablet e244689cef6844e2935becd265eecad0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1a69b6de5d433f82579b1259548fd2) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.791486 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b76d1bbf90c6416ca60805c9d8053bd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e10fe1445be457a8c848c30771af866) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.791641 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet b76d1bbf90c6416ca60805c9d8053bd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e10fe1445be457a8c848c30771af866) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.792025 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b85d63974ab418eb1921dd605fbf2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.791839 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 6942d102069d4c33b23fa2c0b29f94a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:02.793931 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e244689cef6844e2935becd265eecad0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e244689cef6844e2935becd265eecad0 already in progress: creating tablet
I20250115 14:52:02.795094 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b85d63974ab418eb1921dd605fbf2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:02.795787 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c574dba6b2a04915bdd871de4dff4df0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c574dba6b2a04915bdd871de4dff4df0 already in progress: creating tablet
I20250115 14:52:02.795843 739 tablet_service.cc:1514] Processing DeleteTablet for tablet e244689cef6844e2935becd265eecad0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1a69b6de5d433f82579b1259548fd2) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:02.796017 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 5b85d63974ab418eb1921dd605fbf2b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b85d63974ab418eb1921dd605fbf2b0 already in progress: creating tablet
I20250115 14:52:02.796123 745 tablet_service.cc:1514] Processing DeleteTablet for tablet c574dba6b2a04915bdd871de4dff4df0 with delete_type TABLET_DATA_DELETED (replaced by tablet ca5cc55b50ee4953800f10efe0ffb546) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:02.796209 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b76d1bbf90c6416ca60805c9d8053bd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b76d1bbf90c6416ca60805c9d8053bd4 already in progress: creating tablet
I20250115 14:52:02.796343 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b76d1bbf90c6416ca60805c9d8053bd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e10fe1445be457a8c848c30771af866) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:02.796453 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6942d102069d4c33b23fa2c0b29f94a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6942d102069d4c33b23fa2c0b29f94a6 already in progress: creating tablet
I20250115 14:52:02.796533 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 6942d102069d4c33b23fa2c0b29f94a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.797082 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 1f1df9dc743342b4b7bdb1a8d3fcd5b0 (table default.Table_fc422b5daf4a4770945aab3d34bc5d35 [id=c8b8a13b708841e2ad2017a4fa7d01d1]) successfully deleted
I20250115 14:52:02.800403 1838 raft_consensus.cc:695] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:02.800683 1838 consensus_queue.cc:237] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:02.803395 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc9ce226d3ba44d6908257bbd0df3a12 consensus state has no leader
I20250115 14:52:02.803755 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0a8e879b253d48d3b02055cb2bd9663e" } modify_external_catalogs: true
I20250115 14:52:02.803932 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0a8e879b253d48d3b02055cb2bd9663e" } modify_external_catalogs: true
I20250115 14:52:02.805039 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_02969c37d40f489482e5eba20a9e2a8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.805570 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02969c37d40f489482e5eba20a9e2a8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.805624 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_880a5165055d48ffba4f0508b2aebfdb" } new_table_name: "default.Table_ff30997993b446919e3eb0926e313f0f" modify_external_catalogs: true
I20250115 14:52:02.807067 1876 raft_consensus.cc:491] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.807031 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_33d4b270a59a4b438798f77331accb24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:02.807197 1876 raft_consensus.cc:513] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:02.807456 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33d4b270a59a4b438798f77331accb24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.807818 1876 leader_election.cc:290] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:02.808526 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8edddd03b4f34de4bb4801a6dcca183b consensus state has no leader
I20250115 14:52:02.808915 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.809432 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.788264 1166 raft_consensus.cc:2267] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.814900 1166 ts_tablet_manager.cc:1905] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.815271 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc975b044dd439dbd1f8f9a82cce751" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.816151 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc975b044dd439dbd1f8f9a82cce751" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.816599 983 leader_election.cc:343] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:02.816988 984 leader_election.cc:343] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.817077 984 leader_election.cc:304] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.817940 1955 raft_consensus.cc:491] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.819116 1955 raft_consensus.cc:513] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.819607 1876 raft_consensus.cc:2743] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.820353 1955 leader_election.cc:290] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.820458 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f07ef30cac2470d9b2f428338dd37a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.821231 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c984613c7d45d5b7c513cd900c9558" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.821466 913 raft_consensus.cc:2388] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d7ecece2ab49e1be7079216517d13b in current term 1: Already voted for candidate c238c037a1c74725a001e2b4a1ff4304 in this term.
I20250115 14:52:02.821748 879 tablet_service.cc:1514] Processing DeleteTablet for tablet d97c2f443ab040f9bc1924f45379165e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.803752 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2061d08104dd455788c1d0e8c792abc7" } new_table_name: "default.Table_1c51a5aaced84915a1dba864731460fe" modify_external_catalogs: true
W20250115 14:52:02.822588 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b consensus state has no leader
W20250115 14:52:02.822950 710 leader_election.cc:343] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d4c984613c7d45d5b7c513cd900c9558
I20250115 14:52:02.823078 710 leader_election.cc:304] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:02.824576 1168 tablet_bootstrap.cc:492] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:02.825393 1164 tablet_bootstrap.cc:492] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.825595 1164 ts_tablet_manager.cc:1397] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.245s user 0.032s sys 0.060s
I20250115 14:52:02.826216 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4add05103df84280812bf61910d76d1d" } modify_external_catalogs: true
I20250115 14:52:02.826400 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4add05103df84280812bf61910d76d1d" } modify_external_catalogs: true
I20250115 14:52:02.826797 1164 raft_consensus.cc:357] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.826930 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 72a97d102c174bc3a5e3a3c5762033b3
I20250115 14:52:02.827086 1164 raft_consensus.cc:383] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.827209 1164 raft_consensus.cc:738] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.827875 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a97d102c174bc3a5e3a3c5762033b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.827785 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7aef87d795794fd4ae0d9a8776e8cd79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.828353 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7aef87d795794fd4ae0d9a8776e8cd79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:02.829411 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 72a97d102c174bc3a5e3a3c5762033b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a97d102c174bc3a5e3a3c5762033b3 already in progress: creating tablet
I20250115 14:52:02.829627 1876 raft_consensus.cc:491] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.829746 1876 raft_consensus.cc:513] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.831826 1876 leader_election.cc:290] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.831817 1164 consensus_queue.cc:260] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.832367 1164 ts_tablet_manager.cc:1428] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:52:02.832655 1164 tablet_bootstrap.cc:492] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.834529 1164 tablet_bootstrap.cc:654] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.830381 1955 raft_consensus.cc:3054] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:02.830775 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bae96c834a8bbbb1174c027935a2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.831192 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a97d102c174bc3a5e3a3c5762033b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:02.836078 984 leader_election.cc:343] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.838232 1955 raft_consensus.cc:2743] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.829852 1167 ts_tablet_manager.cc:1918] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.830704 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a97d102c174bc3a5e3a3c5762033b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.832417 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bae96c834a8bbbb1174c027935a2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.838528 1168 ts_tablet_manager.cc:1397] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.266s user 0.030s sys 0.037s
W20250115 14:52:02.838922 983 leader_election.cc:343] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.839028 983 leader_election.cc:304] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.839191 1168 raft_consensus.cc:357] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.839346 1168 raft_consensus.cc:383] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.839373 1876 raft_consensus.cc:2743] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.839393 1168 raft_consensus.cc:738] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.839668 1167 log.cc:1198] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f847f92cd122423aa93f1afa701b0bbb
I20250115 14:52:02.840013 1167 ts_tablet_manager.cc:1939] T f847f92cd122423aa93f1afa701b0bbb P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:02.840335 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 461) took 2808 ms. Trace:
I20250115 14:52:02.840454 1167 rpcz_store.cc:276] 0115 14:52:00.032340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.064353 (+ 32013us) service_pool.cc:225] Handling call
0115 14:52:02.775526 (+2711173us) ts_tablet_manager.cc:1145] Deleting tablet f847f92cd122423aa93f1afa701b0bbb
0115 14:52:02.775536 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.779058 (+ 3522us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.829750 (+ 50692us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.840321 (+ 10571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2710500}
I20250115 14:52:02.840910 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 465) took 2740 ms. Trace:
I20250115 14:52:02.841025 1167 rpcz_store.cc:276] 0115 14:52:00.100877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.100918 (+ 41us) service_pool.cc:225] Handling call
0115 14:52:02.840810 (+2739892us) ts_tablet_manager.cc:1145] Deleting tablet 8f8bae9867d745afb1dfdf47bcb881da
0115 14:52:02.840819 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.840898 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2739633}
I20250115 14:52:02.841368 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 466) took 2714 ms. Trace:
I20250115 14:52:02.841463 1167 rpcz_store.cc:276] 0115 14:52:00.126690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.126747 (+ 57us) service_pool.cc:225] Handling call
0115 14:52:02.841284 (+2714537us) ts_tablet_manager.cc:1145] Deleting tablet 4f1ce59e05054a009cd36e12db35d449
0115 14:52:02.841292 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.841357 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2714341}
I20250115 14:52:02.841724 1167 tablet_replica.cc:331] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:02.841919 1167 raft_consensus.cc:2238] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.841957 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f847f92cd122423aa93f1afa701b0bbb (table default.Table_458a573b4a4b4714950723b426f0d265 [id=1fd54f7dafdc4777bd06982391452889]) successfully deleted
W20250115 14:52:02.842200 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 8f8bae9867d745afb1dfdf47bcb881da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f8bae9867d745afb1dfdf47bcb881da already in progress: creating tablet
W20250115 14:52:02.842397 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 4f1ce59e05054a009cd36e12db35d449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1ce59e05054a009cd36e12db35d449 already in progress: creating tablet
I20250115 14:52:02.842406 1167 raft_consensus.cc:2267] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.842839 1167 ts_tablet_manager.cc:1905] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.845046 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_803f17ac9c4e40ffb06dc6ac0faaf7ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:02.845522 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_803f17ac9c4e40ffb06dc6ac0faaf7ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:02.839576 1168 consensus_queue.cc:260] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.846256 1168 ts_tablet_manager.cc:1428] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250115 14:52:02.846632 1168 tablet_bootstrap.cc:492] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:02.847941 1168 tablet_bootstrap.cc:654] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:02.849011 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb consensus state has no leader
I20250115 14:52:02.851172 1162 tablet_bootstrap.cc:492] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.851315 1162 ts_tablet_manager.cc:1397] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.117s user 0.012s sys 0.011s
I20250115 14:52:02.854190 1162 raft_consensus.cc:357] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.854465 1162 raft_consensus.cc:383] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.854565 1162 raft_consensus.cc:738] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.854868 1162 consensus_queue.cc:260] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.855952 1162 ts_tablet_manager.cc:1428] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:02.856251 1162 tablet_bootstrap.cc:492] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.860538 1162 tablet_bootstrap.cc:654] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.861914 1167 ts_tablet_manager.cc:1918] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.862038 1167 log.cc:1198] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d4c984613c7d45d5b7c513cd900c9558
I20250115 14:52:02.862303 1167 ts_tablet_manager.cc:1939] T d4c984613c7d45d5b7c513cd900c9558 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:02.862305 1164 tablet_bootstrap.cc:492] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:02.862538 1164 ts_tablet_manager.cc:1397] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250115 14:52:02.862609 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 479) took 2685 ms. Trace:
I20250115 14:52:02.862684 1167 rpcz_store.cc:276] 0115 14:52:00.176703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.176751 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:02.841689 (+2664938us) ts_tablet_manager.cc:1145] Deleting tablet d4c984613c7d45d5b7c513cd900c9558
0115 14:52:02.841696 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.850746 (+ 9050us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.861814 (+ 11068us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.862592 (+ 778us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2664761}
I20250115 14:52:02.863277 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d4c984613c7d45d5b7c513cd900c9558 (table default.Table_272f9650cb3142f781f1d4d7c4769dfe [id=2fb684a787e4420aa9218b369eba5cf2]) successfully deleted
I20250115 14:52:02.863545 1164 raft_consensus.cc:357] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.863832 1164 raft_consensus.cc:383] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.863893 1164 raft_consensus.cc:738] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.864132 1164 consensus_queue.cc:260] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.865432 1164 ts_tablet_manager.cc:1428] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:02.865872 1164 tablet_bootstrap.cc:492] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:02.866290 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7]). Expected version 2 got 0
I20250115 14:52:02.866537 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet fbf94077e0e343b8938b081102dfd2af (table default.Table_87cf8b44f8fb4fbda80f72656bfaa074 [id=00a89ba1b93f4132aae8fe6e748df77e]). Expected version 1 got 0
I20250115 14:52:02.866683 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 338c0f6f39fd4335bbcda733b28b8bf8 (table default.Table_dd1b72b0118645adb63b44442697414f [id=6756cde43edd4d1e8a2768c101c4aa19]). Expected version 2 got 0
I20250115 14:52:02.866925 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 009ddfe9dfe14ba2b067f907a9d13538 (table default.Table_96eab6cd20b842138b5acc7c8d0ac5d6 [id=816d14738e72406a88b9eed42d72be73]). Expected version 2 got 0
W20250115 14:52:02.867499 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbf94077e0e343b8938b081102dfd2af consensus state has no leader
I20250115 14:52:02.868893 455 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]). Expected version 2 got 0
I20250115 14:52:02.872179 447 catalog_manager.cc:5526] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:02.875111 1167 tablet_replica.cc:331] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:02.875461 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]). Expected version 2 got 0
I20250115 14:52:02.875726 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]). Expected version 1 got 0
I20250115 14:52:02.875988 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2560d42c15c64b4c9c0cfe95706b4d9e
I20250115 14:52:02.876071 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1b2d04209fd34918b83978325a5e1df8
I20250115 14:52:02.876330 1167 raft_consensus.cc:2238] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.876504 1167 raft_consensus.cc:2267] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.878371 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2d04209fd34918b83978325a5e1df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.877822 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 2560d42c15c64b4c9c0cfe95706b4d9e with delete_type TABLET_DATA_DELETED (replaced by tablet f4536bb3d44b40d2825e1f4e59d8a148) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.879148 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2d04209fd34918b83978325a5e1df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.879348 1167 ts_tablet_manager.cc:1905] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:02.879357 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 1b2d04209fd34918b83978325a5e1df8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2d04209fd34918b83978325a5e1df8 already in progress: creating tablet
I20250115 14:52:02.879432 451 catalog_manager.cc:6596] ReplaceTablet: tablet 2560d42c15c64b4c9c0cfe95706b4d9e deleted and replaced by tablet f4536bb3d44b40d2825e1f4e59d8a148
I20250115 14:52:02.879951 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 2560d42c15c64b4c9c0cfe95706b4d9e with delete_type TABLET_DATA_DELETED (replaced by tablet f4536bb3d44b40d2825e1f4e59d8a148) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.880174 1876 raft_consensus.cc:491] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.880324 1876 raft_consensus.cc:513] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.878772 1168 tablet_bootstrap.cc:492] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:02.880681 1168 ts_tablet_manager.cc:1397] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.005s
W20250115 14:52:02.880745 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2560d42c15c64b4c9c0cfe95706b4d9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2560d42c15c64b4c9c0cfe95706b4d9e already in progress: creating tablet
I20250115 14:52:02.880991 1876 leader_election.cc:290] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.880321 1162 tablet_bootstrap.cc:492] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.881410 1162 ts_tablet_manager.cc:1397] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20250115 14:52:02.881381 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13b4c0edc2c4fc9a1c527e1ff6a3d00" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:02.881852 983 leader_election.cc:343] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.882117 1162 raft_consensus.cc:357] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.882282 1162 raft_consensus.cc:383] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.882367 1162 raft_consensus.cc:738] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.882596 1162 consensus_queue.cc:260] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.885046 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13b4c0edc2c4fc9a1c527e1ff6a3d00" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.885548 984 leader_election.cc:343] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.885648 984 leader_election.cc:304] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.885974 1876 raft_consensus.cc:2743] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.887038 1162 ts_tablet_manager.cc:1428] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250115 14:52:02.887367 1162 tablet_bootstrap.cc:492] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:02.888890 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2d04209fd34918b83978325a5e1df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.889142 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2560d42c15c64b4c9c0cfe95706b4d9e with delete_type TABLET_DATA_DELETED (replaced by tablet f4536bb3d44b40d2825e1f4e59d8a148) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.889601 451 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:52:02.889827 451 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:02.891336 1164 tablet_bootstrap.cc:654] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.895581 1162 tablet_bootstrap.cc:654] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.896940 451 master_service.cc:946] ReplaceTablet: received request to replace tablet ff9b57b8b9434f55ace5a4f47380df54 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:02.897876 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fbde011216426cb41c51fe359dcb5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.898555 1168 raft_consensus.cc:357] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.898761 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5265d5eeac83481ca97801e3475886da
I20250115 14:52:02.899722 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 5265d5eeac83481ca97801e3475886da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:02.900779 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 5265d5eeac83481ca97801e3475886da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5265d5eeac83481ca97801e3475886da already in progress: creating tablet
I20250115 14:52:02.902264 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5265d5eeac83481ca97801e3475886da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:02.900400 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 5265d5eeac83481ca97801e3475886da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.903509 1168 raft_consensus.cc:383] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.903745 1168 raft_consensus.cc:738] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.903893 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 010e6535b6bc44f391e6f2b6f1686532 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d891d568574c51b0d062ca9ac77f45) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.904135 1751 tablet_replica.cc:331] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.904155 1168 consensus_queue.cc:260] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.904574 1751 raft_consensus.cc:2238] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.904659 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet e198b4e1a65b4704bf2bb8f7d5987385 with delete_type TABLET_DATA_DELETED (replaced by tablet 65a6f83730f74d13823bd15d93538df7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.904918 1751 raft_consensus.cc:2267] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.905108 1162 tablet_bootstrap.cc:492] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:02.905304 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ad2b7350294375bd30fb4b46906491 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.905439 1162 ts_tablet_manager.cc:1397] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20250115 14:52:02.905972 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc975b044dd439dbd1f8f9a82cce751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.905997 1751 ts_tablet_manager.cc:1905] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.906188 1162 raft_consensus.cc:357] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.906370 1162 raft_consensus.cc:383] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:02.906422 1162 raft_consensus.cc:738] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:02.906592 1162 consensus_queue.cc:260] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.906678 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07686c40324233a49e4c49aeed73f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.907306 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet d71a1ecd900942e5ae788c9a28c70a7a with delete_type TABLET_DATA_DELETED (replaced by tablet 0651be54e0b244a4bace9f47efc9b58a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.907585 1162 ts_tablet_manager.cc:1428] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:02.909031 1168 ts_tablet_manager.cc:1428] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250115 14:52:02.909694 1162 tablet_bootstrap.cc:492] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:02.909142 847 consensus_peers.cc:487] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9145bbaf3e234f079e355454f3ce28cb. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:02.910255 1168 tablet_bootstrap.cc:492] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:02.911511 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cba5cc51d7414a9352caedcc362c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.912513 1983 tablet.cc:1721] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:02.914709 1982 tablet.cc:1721] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:02.918853 1167 ts_tablet_manager.cc:1918] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:02.918949 1167 log.cc:1198] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d5a5a92804cb462181e68e1cec02102b
I20250115 14:52:02.919265 1167 ts_tablet_manager.cc:1939] T d5a5a92804cb462181e68e1cec02102b P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:02.919659 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet be63bae96c834a8bbbb1174c027935a2 (table default.Table_f85e432eb37548eb9d03942617305020 [id=b2c6ee1ec113494cac7d1bf11a928714]). Expected version 2 got 0
I20250115 14:52:02.919721 741 tablet_service.cc:1467] Processing CreateTablet for tablet 74d00bcee22d40769102eacaf728480d (DEFAULT_TABLE table=default.Table_9af73c3d469c4f2099396a0d51abc8ea [id=1ccab9d1b41c4795b17a4bf435ece517]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.919865 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_1ff8a32badfe4ea8997fac383c0e5559 [id=30b975fd37374a5a90d8cbda734489f1]). Expected version 2 got 0
I20250115 14:52:02.919956 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 478) took 2743 ms. Trace:
I20250115 14:52:02.920102 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d00bcee22d40769102eacaf728480d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.920050 1167 rpcz_store.cc:276] 0115 14:52:00.176448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.177567 (+ 1119us) service_pool.cc:225] Handling call
0115 14:52:02.875049 (+2697482us) ts_tablet_manager.cc:1145] Deleting tablet d5a5a92804cb462181e68e1cec02102b
0115 14:52:02.875063 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.916044 (+ 40981us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.918776 (+ 2732us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:02.919938 (+ 1162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2697291}
I20250115 14:52:02.911908 1162 tablet_bootstrap.cc:654] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:02.924348 751 tablet_service.cc:1467] Processing CreateTablet for tablet 655489ad9bd34be0a8283a295f583092 (DEFAULT_TABLE table=default.Table_bbcbd9770e46481a894a5352459d371d [id=4dcefc85053b4488bc42193abc4ac607]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.924712 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 655489ad9bd34be0a8283a295f583092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.925303 879 tablet_service.cc:1467] Processing CreateTablet for tablet 655489ad9bd34be0a8283a295f583092 (DEFAULT_TABLE table=default.Table_bbcbd9770e46481a894a5352459d371d [id=4dcefc85053b4488bc42193abc4ac607]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.925829 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 655489ad9bd34be0a8283a295f583092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.927688 882 tablet_service.cc:1467] Processing CreateTablet for tablet 74d00bcee22d40769102eacaf728480d (DEFAULT_TABLE table=default.Table_9af73c3d469c4f2099396a0d51abc8ea [id=1ccab9d1b41c4795b17a4bf435ece517]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.927989 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d00bcee22d40769102eacaf728480d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.929857 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 655489ad9bd34be0a8283a295f583092 (DEFAULT_TABLE table=default.Table_bbcbd9770e46481a894a5352459d371d [id=4dcefc85053b4488bc42193abc4ac607]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.930186 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 655489ad9bd34be0a8283a295f583092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.932686 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d5a5a92804cb462181e68e1cec02102b (table default.Table_c0256da787104a42a1bc1e3946e11fd6 [id=22c418c3ac6648c687a8ae14d439aea5]) successfully deleted
I20250115 14:52:02.934422 1167 tablet_replica.cc:331] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:02.936604 1168 tablet_bootstrap.cc:654] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:02.936606 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 52265.1us
I20250115 14:52:02.939898 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:02.940851 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet caa1827366a34f039fd12deb855f07b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.940985 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 104282a09c024bbaafeb5f28f91dadcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:02.942925 1167 raft_consensus.cc:2238] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.943233 1876 raft_consensus.cc:491] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.943352 1876 raft_consensus.cc:513] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.943953 1876 leader_election.cc:290] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.944571 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338c0f6f39fd4335bbcda733b28b8bf8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:02.945026 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338c0f6f39fd4335bbcda733b28b8bf8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:02.945441 1167 raft_consensus.cc:2267] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:02.945465 983 leader_election.cc:343] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:02.945657 984 leader_election.cc:343] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.945750 984 leader_election.cc:304] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.946154 1876 raft_consensus.cc:2743] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.946768 1167 ts_tablet_manager.cc:1905] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.947547 1876 raft_consensus.cc:491] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.947664 1876 raft_consensus.cc:513] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:02.948228 1876 leader_election.cc:290] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:02.948721 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171643c40c9b43ffb823f7c33789284d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:02.949501 983 leader_election.cc:343] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.950026 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171643c40c9b43ffb823f7c33789284d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:02.950485 984 leader_election.cc:343] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:02.950573 984 leader_election.cc:304] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:02.950942 1876 raft_consensus.cc:2743] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:02.951922 878 tablet_service.cc:1467] Processing CreateTablet for tablet 1a543188b2ca4478b79d4ad08ee30b89 (DEFAULT_TABLE table=default.Table_4b828508be354eb4a8f42f5ba4b2e760 [id=4b7f42edeb5e4550b2d4d738757a42d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.952283 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a543188b2ca4478b79d4ad08ee30b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.952204 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 74d00bcee22d40769102eacaf728480d (DEFAULT_TABLE table=default.Table_9af73c3d469c4f2099396a0d51abc8ea [id=1ccab9d1b41c4795b17a4bf435ece517]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.952550 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d00bcee22d40769102eacaf728480d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.952520 741 tablet_service.cc:1467] Processing CreateTablet for tablet 2b26b9b1caf748aba7d36a04fbba2a80 (DEFAULT_TABLE table=default.Table_1fe291920627495bb359337954580862 [id=04ac31dfe5d649b587e28c62958077bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.952870 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b26b9b1caf748aba7d36a04fbba2a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.953788 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1a543188b2ca4478b79d4ad08ee30b89 (DEFAULT_TABLE table=default.Table_4b828508be354eb4a8f42f5ba4b2e760 [id=4b7f42edeb5e4550b2d4d738757a42d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.954077 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a543188b2ca4478b79d4ad08ee30b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.946949 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb] alter complete (version 1)
I20250115 14:52:02.956104 883 tablet_service.cc:1514] Processing DeleteTablet for tablet e3fbde011216426cb41c51fe359dcb5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:54 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:02.956506 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 2b26b9b1caf748aba7d36a04fbba2a80 (DEFAULT_TABLE table=default.Table_1fe291920627495bb359337954580862 [id=04ac31dfe5d649b587e28c62958077bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.956835 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b26b9b1caf748aba7d36a04fbba2a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.960404 1751 ts_tablet_manager.cc:1918] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.960505 1751 log.cc:1198] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/010e6535b6bc44f391e6f2b6f1686532
I20250115 14:52:02.960794 1751 ts_tablet_manager.cc:1939] T 010e6535b6bc44f391e6f2b6f1686532 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:02.961139 882 tablet_service.cc:1467] Processing CreateTablet for tablet 2b26b9b1caf748aba7d36a04fbba2a80 (DEFAULT_TABLE table=default.Table_1fe291920627495bb359337954580862 [id=04ac31dfe5d649b587e28c62958077bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.961522 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b26b9b1caf748aba7d36a04fbba2a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.965003 745 tablet_service.cc:1467] Processing CreateTablet for tablet 929830d8e8fd4b0c8a7b18c941706929 (DEFAULT_TABLE table=default.Table_9c44f30ee706467f8ae6014dd1b51473 [id=b2edb82a004c4fdba6847b27f6c51007]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.965366 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 929830d8e8fd4b0c8a7b18c941706929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.966990 878 tablet_service.cc:1467] Processing CreateTablet for tablet c24df374316a4032bfb88c89b7c78106 (DEFAULT_TABLE table=default.Table_bf35b2839ef143e4b6aeec7caf9313ce [id=fce4b0011ae44fb5928471814b69bfc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.967350 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24df374316a4032bfb88c89b7c78106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.968538 739 tablet_service.cc:1467] Processing CreateTablet for tablet c24df374316a4032bfb88c89b7c78106 (DEFAULT_TABLE table=default.Table_bf35b2839ef143e4b6aeec7caf9313ce [id=fce4b0011ae44fb5928471814b69bfc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.968900 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24df374316a4032bfb88c89b7c78106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.971241 1166 ts_tablet_manager.cc:1918] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:02.973489 883 tablet_service.cc:1467] Processing CreateTablet for tablet e8dfc6349ccd4cf1a72c91bd7e22250c (DEFAULT_TABLE table=default.Table_c699f7ff601546bb985cc2e19af86edb [id=39cbe4bf57d04c3c81fa0cf35be6ba2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.973796 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8dfc6349ccd4cf1a72c91bd7e22250c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.973747 752 tablet_service.cc:1467] Processing CreateTablet for tablet e8dfc6349ccd4cf1a72c91bd7e22250c (DEFAULT_TABLE table=default.Table_c699f7ff601546bb985cc2e19af86edb [id=39cbe4bf57d04c3c81fa0cf35be6ba2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.974083 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8dfc6349ccd4cf1a72c91bd7e22250c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.985443 1166 log.cc:1198] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9145bbaf3e234f079e355454f3ce28cb
I20250115 14:52:02.985591 879 tablet_service.cc:1467] Processing CreateTablet for tablet 929830d8e8fd4b0c8a7b18c941706929 (DEFAULT_TABLE table=default.Table_9c44f30ee706467f8ae6014dd1b51473 [id=b2edb82a004c4fdba6847b27f6c51007]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.985812 1166 ts_tablet_manager.cc:1939] T 9145bbaf3e234f079e355454f3ce28cb P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:02.985903 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 929830d8e8fd4b0c8a7b18c941706929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.986666 1166 tablet_replica.cc:331] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:02.986879 1166 raft_consensus.cc:2238] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:02.987053 1166 raft_consensus.cc:2267] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:02.987542 1166 ts_tablet_manager.cc:1905] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:02.991866 1751 tablet_replica.cc:331] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:02.964566 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 1a543188b2ca4478b79d4ad08ee30b89 (DEFAULT_TABLE table=default.Table_4b828508be354eb4a8f42f5ba4b2e760 [id=4b7f42edeb5e4550b2d4d738757a42d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.992403 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a543188b2ca4478b79d4ad08ee30b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:02.992642 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 010e6535b6bc44f391e6f2b6f1686532 (table default.Table_c5f747dc2f6b4cffb7b791ea6456c5b3 [id=13d88a401bed452688d5bb6f4dcfd70b]) successfully deleted
I20250115 14:52:02.965780 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 929830d8e8fd4b0c8a7b18c941706929 (DEFAULT_TABLE table=default.Table_9c44f30ee706467f8ae6014dd1b51473 [id=b2edb82a004c4fdba6847b27f6c51007]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.996845 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 929830d8e8fd4b0c8a7b18c941706929. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.003675 1167 ts_tablet_manager.cc:1918] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:03.003892 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9145bbaf3e234f079e355454f3ce28cb (table default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]) successfully deleted
I20250115 14:52:03.004671 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dbb5dd7ec474409b9743a74e69cc1b44" } modify_external_catalogs: true
I20250115 14:52:03.004822 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dbb5dd7ec474409b9743a74e69cc1b44" } modify_external_catalogs: true
I20250115 14:52:03.004616 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e1110e8f11bb4423bcdd8fedb129e390" } modify_external_catalogs: true
I20250115 14:52:03.005640 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e1110e8f11bb4423bcdd8fedb129e390" } modify_external_catalogs: true
I20250115 14:52:03.006523 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5ef75e2dec224af78b8b5da8cf17abfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.007099 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ef75e2dec224af78b8b5da8cf17abfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.009224 1167 log.cc:1198] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/4239747374aa4d6d866961430786677c
I20250115 14:52:03.010020 1167 ts_tablet_manager.cc:1939] T 4239747374aa4d6d866961430786677c P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:03.011479 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 480) took 2830 ms. Trace:
I20250115 14:52:03.012585 1167 rpcz_store.cc:276] 0115 14:52:00.181238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.181286 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:02.934376 (+2753090us) ts_tablet_manager.cc:1145] Deleting tablet 4239747374aa4d6d866961430786677c
0115 14:52:02.934386 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:02.989832 (+ 55446us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.003603 (+ 13771us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.011464 (+ 7861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2752891}
I20250115 14:52:03.013908 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 481) took 2828 ms. Trace:
I20250115 14:52:03.015058 1167 rpcz_store.cc:276] 0115 14:52:00.185162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.185211 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:03.013795 (+2828584us) ts_tablet_manager.cc:1145] Deleting tablet 3a0d97fa57484b9d8cb879fedda61612
0115 14:52:03.013804 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.013893 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2828429}
I20250115 14:52:03.016168 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5c73eff7b16a4a11ad47cd4c32a6fd0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.016430 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 4239747374aa4d6d866961430786677c (table default.Table_8e6652dab3bc46b7b4f1b66415398217 [id=8941c6c27037475990cf318614854a27]) successfully deleted
W20250115 14:52:03.016714 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c73eff7b16a4a11ad47cd4c32a6fd0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:03.016876 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3a0d97fa57484b9d8cb879fedda61612 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a0d97fa57484b9d8cb879fedda61612 already in progress: creating tablet
I20250115 14:52:03.016368 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e48cf46cf80d4d98a4955f201d993354" } new_table_name: "default.Table_ca5ad5415b9144e583b4c771d24d4431" modify_external_catalogs: true
I20250115 14:52:03.018082 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d03b53f6993c4f3aae49c2155890bfc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.018354 1167 tablet_replica.cc:331] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:03.018520 1167 raft_consensus.cc:2238] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
W20250115 14:52:03.018568 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d03b53f6993c4f3aae49c2155890bfc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.018826 1167 raft_consensus.cc:2267] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.019337 1167 ts_tablet_manager.cc:1905] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.019574 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8234c8fb043a45b495995ce2bcd14cc7" } new_table_name: "default.Table_d743d5b92e184c949a71cff40336acf9" modify_external_catalogs: true
I20250115 14:52:03.020535 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e5a059e2f0654a4aafe9e8082b8fde7a
I20250115 14:52:03.020735 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0b2109f1c1814a63a365822b2102e61e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.021220 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b2109f1c1814a63a365822b2102e61e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.021613 878 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a059e2f0654a4aafe9e8082b8fde7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.021844 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a059e2f0654a4aafe9e8082b8fde7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.021960 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ba596cfb3ce64718a42a9083975edc8d
I20250115 14:52:03.022578 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a059e2f0654a4aafe9e8082b8fde7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.022827 878 tablet_service.cc:1514] Processing DeleteTablet for tablet ba596cfb3ce64718a42a9083975edc8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.023242 752 tablet_service.cc:1514] Processing DeleteTablet for tablet ba596cfb3ce64718a42a9083975edc8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.026376 1168 tablet_bootstrap.cc:492] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.027541 1168 ts_tablet_manager.cc:1397] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.117s user 0.008s sys 0.007s
I20250115 14:52:03.028736 1168 raft_consensus.cc:357] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.031002 1168 raft_consensus.cc:383] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.031282 1168 raft_consensus.cc:738] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.031705 1168 consensus_queue.cc:260] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.033581 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet ba596cfb3ce64718a42a9083975edc8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:03.035885 848 consensus_peers.cc:487] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd4102b1f83e414d93a6a68315e36d16. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:03.038249 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f85e432eb37548eb9d03942617305020" } new_table_name: "default.Table_705745b616194094bb03915f2f651122" modify_external_catalogs: true
I20250115 14:52:03.039959 1162 tablet_bootstrap.cc:492] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.040397 1168 ts_tablet_manager.cc:1428] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250115 14:52:03.042517 1162 ts_tablet_manager.cc:1397] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.133s user 0.001s sys 0.017s
I20250115 14:52:03.042517 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c385edf4414157845b971aaf366831 with delete_type TABLET_DATA_DELETED (replaced by tablet 84844b5f50af408ab7dda73f7421cd9d) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.040730 1168 tablet_bootstrap.cc:492] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:03.043395 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ded663ac1197454bbde3c4c2d526002d consensus state has no leader
I20250115 14:52:03.044629 1162 raft_consensus.cc:357] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.044819 1162 raft_consensus.cc:383] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.044875 1162 raft_consensus.cc:738] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.044906 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_00e45525259140518db638102c114d02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.045085 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8b9b25dd8a85434a83d2641ffbc5189e" } modify_external_catalogs: true
W20250115 14:52:03.045456 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00e45525259140518db638102c114d02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.045459 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8b9b25dd8a85434a83d2641ffbc5189e" } modify_external_catalogs: true
I20250115 14:52:03.046725 1168 tablet_bootstrap.cc:654] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.048463 1162 consensus_queue.cc:260] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:02.972227 1019 tablet_service.cc:1467] Processing CreateTablet for tablet c24df374316a4032bfb88c89b7c78106 (DEFAULT_TABLE table=default.Table_bf35b2839ef143e4b6aeec7caf9313ce [id=fce4b0011ae44fb5928471814b69bfc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:02.972496 1876 raft_consensus.cc:491] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:02.977162 1016 tablet_service.cc:1467] Processing CreateTablet for tablet e8dfc6349ccd4cf1a72c91bd7e22250c (DEFAULT_TABLE table=default.Table_c699f7ff601546bb985cc2e19af86edb [id=39cbe4bf57d04c3c81fa0cf35be6ba2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.050153 1876 raft_consensus.cc:513] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.050410 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8dfc6349ccd4cf1a72c91bd7e22250c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.050758 1876 leader_election.cc:290] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.050985 1751 raft_consensus.cc:2238] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.051262 1751 raft_consensus.cc:2267] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.051771 1751 ts_tablet_manager.cc:1905] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.055785 1751 ts_tablet_manager.cc:1918] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.055877 1751 log.cc:1198] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e198b4e1a65b4704bf2bb8f7d5987385
I20250115 14:52:03.056103 1751 ts_tablet_manager.cc:1939] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:03.057132 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e198b4e1a65b4704bf2bb8f7d5987385" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.057278 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e198b4e1a65b4704bf2bb8f7d5987385" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.057675 1751 tablet_replica.cc:331] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:03.057839 1751 raft_consensus.cc:2238] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:03.057821 983 leader_election.cc:343] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:03.058161 984 leader_election.cc:343] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.058254 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c24df374316a4032bfb88c89b7c78106. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.058259 984 leader_election.cc:304] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:03.058926 984 raft_consensus.cc:2681] T e198b4e1a65b4704bf2bb8f7d5987385 P c238c037a1c74725a001e2b4a1ff4304: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:52:03.059832 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e198b4e1a65b4704bf2bb8f7d5987385 (table default.Table_efb7e019ddea47a8954081c1715654e3 [id=a3bf28673dac4ddab579709d6fd8885d]) successfully deleted
I20250115 14:52:03.057981 1751 raft_consensus.cc:2267] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.060597 1751 ts_tablet_manager.cc:1905] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.061067 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_41d5e114b3da4fca97f9efc5a69dfe2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.061571 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41d5e114b3da4fca97f9efc5a69dfe2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.065438 1162 ts_tablet_manager.cc:1428] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250115 14:52:03.065943 1167 ts_tablet_manager.cc:1918] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:03.066514 1162 tablet_bootstrap.cc:492] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.066648 1167 log.cc:1198] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/4b90205ef15f470a99c013e7f2cf99c6
I20250115 14:52:03.067245 1167 ts_tablet_manager.cc:1939] T 4b90205ef15f470a99c013e7f2cf99c6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:03.068342 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 486) took 2731 ms. Trace:
W20250115 14:52:03.069886 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 580742eb5dac4dba9995809890e0e315 (table default.Table_d8d3e6a881ce475a8f1319238ab8c610 [id=b6de0dbbced44c83af5f98a9d1020cfc], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 580742eb5dac4dba9995809890e0e315 consensus state has no leader
I20250115 14:52:03.070446 1167 rpcz_store.cc:276] 0115 14:52:00.337291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.337339 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:03.016644 (+2679305us) ts_tablet_manager.cc:1145] Deleting tablet 4b90205ef15f470a99c013e7f2cf99c6
0115 14:52:03.016653 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.034490 (+ 17837us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.065835 (+ 31345us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.068327 (+ 2492us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2670450}
I20250115 14:52:03.071141 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 4b90205ef15f470a99c013e7f2cf99c6 (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]) successfully deleted
I20250115 14:52:03.071401 1751 ts_tablet_manager.cc:1918] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.071556 1751 log.cc:1198] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/37ad2b7350294375bd30fb4b46906491
I20250115 14:52:03.071578 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 502) took 2672 ms. Trace:
I20250115 14:52:03.071863 1751 ts_tablet_manager.cc:1939] T 37ad2b7350294375bd30fb4b46906491 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:03.071947 1167 rpcz_store.cc:276] 0115 14:52:00.399336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.579338 (+180002us) service_pool.cc:225] Handling call
0115 14:52:03.071461 (+2492123us) ts_tablet_manager.cc:1145] Deleting tablet 4f89ec0a5d68408c897f43ab39df62b1
0115 14:52:03.071470 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.071562 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2450011}
W20250115 14:52:03.072585 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 4f89ec0a5d68408c897f43ab39df62b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f89ec0a5d68408c897f43ab39df62b1 already in progress: creating tablet
I20250115 14:52:03.068544 1168 tablet_bootstrap.cc:492] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.072948 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 37ad2b7350294375bd30fb4b46906491 (table default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]) successfully deleted
I20250115 14:52:03.072999 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 501) took 2673 ms. Trace:
I20250115 14:52:03.073122 1168 ts_tablet_manager.cc:1397] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20250115 14:52:03.073272 1751 tablet_replica.cc:331] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:03.073062 1167 rpcz_store.cc:276] 0115 14:52:00.399263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.579225 (+179962us) service_pool.cc:225] Handling call
0115 14:52:03.072888 (+2493663us) ts_tablet_manager.cc:1145] Deleting tablet 4adf1353bd554d8294a7f570931095d2
0115 14:52:03.072897 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.072983 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2375181}
W20250115 14:52:03.073905 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 4adf1353bd554d8294a7f570931095d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4adf1353bd554d8294a7f570931095d2 already in progress: creating tablet
I20250115 14:52:03.073925 1168 raft_consensus.cc:357] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.074090 1168 raft_consensus.cc:383] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.074183 1168 raft_consensus.cc:738] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.074405 1168 consensus_queue.cc:260] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.075145 1751 raft_consensus.cc:2238] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.075459 1168 ts_tablet_manager.cc:1428] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:03.075735 1751 raft_consensus.cc:2267] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.075981 1168 tablet_bootstrap.cc:492] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:03.076885 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 511) took 2522 ms. Trace:
I20250115 14:52:03.077179 1167 rpcz_store.cc:276] 0115 14:52:00.554449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.939753 (+385304us) service_pool.cc:225] Handling call
0115 14:52:03.076786 (+2137033us) ts_tablet_manager.cc:1145] Deleting tablet 9f4475441e4a467a8d593b742d2a5b80
0115 14:52:03.076795 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.076870 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2134293}
I20250115 14:52:03.077179 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0195de43d08f499d99237ab07b24b892"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.077785 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9f4475441e4a467a8d593b742d2a5b80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f4475441e4a467a8d593b742d2a5b80 already in progress: creating tablet
W20250115 14:52:03.077811 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0195de43d08f499d99237ab07b24b892 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.078140 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 509) took 2528 ms. Trace:
I20250115 14:52:03.079757 1168 tablet_bootstrap.cc:654] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.080159 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9c613e4019394b9790eb4d917dc249c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.080371 1167 rpcz_store.cc:276] 0115 14:52:00.549341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.923878 (+374537us) service_pool.cc:225] Handling call
0115 14:52:03.078047 (+2154169us) ts_tablet_manager.cc:1145] Deleting tablet e034d7761a5247c79414b7112a1c97b2
0115 14:52:03.078056 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.078125 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2109075}
W20250115 14:52:03.080652 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c613e4019394b9790eb4d917dc249c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:03.080760 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e034d7761a5247c79414b7112a1c97b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e034d7761a5247c79414b7112a1c97b2 already in progress: creating tablet
I20250115 14:52:03.081171 1167 tablet_replica.cc:331] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:03.081840 1167 raft_consensus.cc:2238] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:03.082422 1167 pending_rounds.cc:62] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:03.082734 1167 pending_rounds.cc:69] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558349409771520 op_type: NO_OP noop_request { }
I20250115 14:52:03.083022 1751 ts_tablet_manager.cc:1905] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.083026 1167 raft_consensus.cc:2883] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:03.083285 1167 raft_consensus.cc:2267] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.084029 1167 ts_tablet_manager.cc:1905] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.084652 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e32df962655644d4b4a7e3ba02623ed3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.085078 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e32df962655644d4b4a7e3ba02623ed3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.087409 1168 tablet_bootstrap.cc:492] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
W20250115 14:52:03.088565 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7718729182a340fe9b2dfea5a8321c1f consensus state has no leader
I20250115 14:52:03.089692 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d18e82c7d46c4394b6037413cd35cef6" } new_table_name: "default.Table_0636a1492a2542d99c65b5d1436dd981" modify_external_catalogs: true
I20250115 14:52:03.092501 1168 ts_tablet_manager.cc:1397] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
W20250115 14:52:03.092521 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be63bae96c834a8bbbb1174c027935a2 consensus state has no leader
I20250115 14:52:03.095188 2043 raft_consensus.cc:491] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.095358 2043 raft_consensus.cc:513] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.093946 1168 raft_consensus.cc:357] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.095602 1168 raft_consensus.cc:383] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.095778 1168 raft_consensus.cc:738] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.096087 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 4934bdde6023419fafd750920bcfe3d4 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:03.096076 1168 consensus_queue.cc:260] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.096416 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104282a09c024bbaafeb5f28f91dadcb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.096910 984 leader_election.cc:343] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.096987 1168 ts_tablet_manager.cc:1428] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:52:03.097452 2043 leader_election.cc:290] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.097353 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4407280ad4d5424b95ea7687acf680ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.097558 1168 tablet_bootstrap.cc:492] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:03.097831 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4407280ad4d5424b95ea7687acf680ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.098361 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104282a09c024bbaafeb5f28f91dadcb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:03.098820 983 leader_election.cc:343] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.098914 983 leader_election.cc:304] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.099545 2043 raft_consensus.cc:2743] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.100664 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_4b79de2dd7fe493f84a77ae3c477e2c8 [id=471b029f1dba4dd087deb339bba02013]). Expected version 1 got 0
I20250115 14:52:03.101346 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dd1b72b0118645adb63b44442697414f" } new_table_name: "default.Table_f367ac1a82704324a73f7079165a6e45" modify_external_catalogs: true
I20250115 14:52:03.102480 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a3a7a487a42949bd9641c38d6c1fcfd3
I20250115 14:52:03.103118 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_609197916c9b4c81aebdaf70e78cc436" } modify_external_catalogs: true
I20250115 14:52:03.103263 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_609197916c9b4c81aebdaf70e78cc436" } modify_external_catalogs: true
I20250115 14:52:03.104529 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a7a487a42949bd9641c38d6c1fcfd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.105016 882 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a7a487a42949bd9641c38d6c1fcfd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.105397 751 tablet_service.cc:1514] Processing DeleteTablet for tablet a3a7a487a42949bd9641c38d6c1fcfd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.106840 1168 tablet_bootstrap.cc:654] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.074906 1162 tablet_bootstrap.cc:654] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.113668 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet caed629878594954842dfc6a40cc8b6d (table default.Table_544d606cf9af4accb670d91d7d717948 [id=b6aeb933c5e64e88a6d794a98650acbe]). Expected version 1 got 0
I20250115 14:52:03.114727 1751 ts_tablet_manager.cc:1918] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.114784 882 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8337d181d44ac7bb82551a677b6831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.115182 1167 ts_tablet_manager.cc:1918] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:03.115384 1751 log.cc:1198] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/5bc975b044dd439dbd1f8f9a82cce751
I20250115 14:52:03.115581 1167 log.cc:1198] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9145bbaf3e234f079e355454f3ce28cb
I20250115 14:52:03.115746 1751 ts_tablet_manager.cc:1939] T 5bc975b044dd439dbd1f8f9a82cce751 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:03.115859 450 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_4b79de2dd7fe493f84a77ae3c477e2c8 [id=471b029f1dba4dd087deb339bba02013]). Expected version 1 got 0
I20250115 14:52:03.116020 1167 ts_tablet_manager.cc:1939] T 9145bbaf3e234f079e355454f3ce28cb P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:03.116583 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a2eb9b62487c4908880b44b9915e4cf7" } modify_external_catalogs: true
I20250115 14:52:03.116739 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a2eb9b62487c4908880b44b9915e4cf7" } modify_external_catalogs: true
I20250115 14:52:03.117054 1843 raft_consensus.cc:491] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.117178 1843 raft_consensus.cc:513] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.117051 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 510) took 2564 ms. Trace:
I20250115 14:52:03.117053 1168 tablet_bootstrap.cc:492] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.117543 1168 ts_tablet_manager.cc:1397] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20250115 14:52:03.117362 1167 rpcz_store.cc:276] 0115 14:52:00.552122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:00.939526 (+387404us) service_pool.cc:225] Handling call
0115 14:52:03.081132 (+2141606us) ts_tablet_manager.cc:1145] Deleting tablet 9145bbaf3e234f079e355454f3ce28cb
0115 14:52:03.081140 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.093651 (+ 12511us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.115101 (+ 21450us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.117036 (+ 1935us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2083683}
I20250115 14:52:03.117769 1843 leader_election.cc:290] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.117841 1167 tablet_replica.cc:331] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:03.118140 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9145bbaf3e234f079e355454f3ce28cb (table default.Table_f7332a834b6a46018bacf978fc59830b [id=a5dc62c04f1747128d36f3baf75a6fab]) successfully deleted
I20250115 14:52:03.118470 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cba5cc51d7414a9352caedcc362c9c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.118891 1167 raft_consensus.cc:2238] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
W20250115 14:52:03.119379 848 leader_election.cc:343] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:03.119513 847 leader_election.cc:343] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 23cba5cc51d7414a9352caedcc362c9c
I20250115 14:52:03.119616 847 leader_election.cc:304] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.119379 1167 raft_consensus.cc:2267] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.120067 1843 raft_consensus.cc:2743] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.120324 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 5bc975b044dd439dbd1f8f9a82cce751 (table default.Table_124bdb85e4194ff5a214a6b79870a1cf [id=f8ea853486b54a39843ef2b5628fbe37]) successfully deleted
I20250115 14:52:03.120527 1751 tablet_replica.cc:331] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:03.120877 1751 raft_consensus.cc:2238] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.121212 1167 ts_tablet_manager.cc:1905] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.121253 1751 raft_consensus.cc:2267] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.122202 1751 ts_tablet_manager.cc:1905] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.125968 1168 raft_consensus.cc:357] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.126708 1168 raft_consensus.cc:383] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.126942 1168 raft_consensus.cc:738] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.127698 1168 consensus_queue.cc:260] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.129184 2041 tablet.cc:1721] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:03.139496 1751 ts_tablet_manager.cc:1918] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.142264 1167 ts_tablet_manager.cc:1918] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:03.149523 1164 tablet_bootstrap.cc:492] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:03.149709 1164 ts_tablet_manager.cc:1397] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.284s user 0.033s sys 0.031s
I20250115 14:52:03.150525 1164 raft_consensus.cc:357] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.150766 1164 raft_consensus.cc:383] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.150864 1164 raft_consensus.cc:738] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.151118 1164 consensus_queue.cc:260] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.151254 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7]). Expected version 2 got 0
I20250115 14:52:03.151691 1167 log.cc:1198] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f8c81f47cd6e4a308cd62124db38d42f
I20250115 14:52:03.151803 1164 ts_tablet_manager.cc:1428] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:03.152063 1167 ts_tablet_manager.cc:1939] T f8c81f47cd6e4a308cd62124db38d42f P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
W20250115 14:52:03.152352 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6203efb862ea4962a6d09f3e4306a92f consensus state has no leader
I20250115 14:52:03.152366 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 518) took 2333 ms. Trace:
I20250115 14:52:03.152438 1167 rpcz_store.cc:276] 0115 14:52:00.818759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.061115 (+242356us) service_pool.cc:225] Handling call
0115 14:52:03.117794 (+2056679us) ts_tablet_manager.cc:1145] Deleting tablet f8c81f47cd6e4a308cd62124db38d42f
0115 14:52:03.117803 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.135809 (+ 18006us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.142185 (+ 6376us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.152351 (+ 10166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2051850}
I20250115 14:52:03.142530 1168 ts_tablet_manager.cc:1428] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250115 14:52:03.152839 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f8c81f47cd6e4a308cd62124db38d42f (table default.Table_d8d3e6a881ce475a8f1319238ab8c610 [id=b6de0dbbced44c83af5f98a9d1020cfc]) successfully deleted
I20250115 14:52:03.142936 1162 tablet_bootstrap.cc:492] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.153115 1162 ts_tablet_manager.cc:1397] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.087s user 0.001s sys 0.006s
I20250115 14:52:03.153797 1162 raft_consensus.cc:357] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.153965 1162 raft_consensus.cc:383] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.154031 1162 raft_consensus.cc:738] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.154238 1162 consensus_queue.cc:260] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.154893 1162 ts_tablet_manager.cc:1428] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:03.155016 1751 log.cc:1198] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/7d07686c40324233a49e4c49aeed73f6
I20250115 14:52:03.155498 1162 tablet_bootstrap.cc:492] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.157490 1168 tablet_bootstrap.cc:492] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:03.158118 1162 tablet_bootstrap.cc:654] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.161648 1168 tablet_bootstrap.cc:654] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.161931 1164 tablet_bootstrap.cc:492] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.162686 1751 ts_tablet_manager.cc:1939] T 7d07686c40324233a49e4c49aeed73f6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:03.163218 1843 raft_consensus.cc:491] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.163331 1843 raft_consensus.cc:513] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.163677 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 7d07686c40324233a49e4c49aeed73f6 (table default.Table_8f5c258dbdc04de79751c3a1b995e0fb [id=e71b0f487dc0428aaa03b4cc7cf95aa0]) successfully deleted
I20250115 14:52:03.163780 751 tablet_service.cc:1514] Processing DeleteTablet for tablet b92231509c754e8dbfc564813a76a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.163952 1843 leader_election.cc:290] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.163967 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7]). Expected version 2 got 0
I20250115 14:52:03.164528 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fbde011216426cb41c51fe359dcb5f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.164762 775 raft_consensus.cc:2463] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:03.165225 848 leader_election.cc:304] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:03.165542 1843 raft_consensus.cc:2798] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.165612 1843 raft_consensus.cc:491] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.165668 1843 raft_consensus.cc:3054] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.165885 1164 tablet_bootstrap.cc:654] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.166325 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 517) took 2350 ms. Trace:
I20250115 14:52:03.166401 1167 rpcz_store.cc:276] 0115 14:52:00.816370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.060986 (+244616us) service_pool.cc:225] Handling call
0115 14:52:03.166179 (+2105193us) ts_tablet_manager.cc:1145] Deleting tablet 759a4ebe7b0444128ddd31a78aad463e
0115 14:52:03.166195 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.166307 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2088859}
W20250115 14:52:03.166916 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 759a4ebe7b0444128ddd31a78aad463e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 759a4ebe7b0444128ddd31a78aad463e already in progress: creating tablet
W20250115 14:52:03.167042 847 leader_election.cc:343] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e3fbde011216426cb41c51fe359dcb5f
I20250115 14:52:03.167348 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 with delete_type TABLET_DATA_DELETED (replaced by tablet cd8586f571634e678e783e32d34f887f) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.167980 1751 tablet_replica.cc:331] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:03.168427 1751 raft_consensus.cc:2238] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.168823 1843 raft_consensus.cc:513] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.169174 1751 raft_consensus.cc:2267] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.169462 1843 leader_election.cc:290] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.169814 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3fbde011216426cb41c51fe359dcb5f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:03.170017 775 raft_consensus.cc:3054] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.171505 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4934bdde6023419fafd750920bcfe3d4
I20250115 14:52:03.171926 449 catalog_manager.cc:6596] ReplaceTablet: tablet 4934bdde6023419fafd750920bcfe3d4 deleted and replaced by tablet 105b6148ae944e5c906e07d987177cc5
I20250115 14:52:03.173151 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4934bdde6023419fafd750920bcfe3d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 105b6148ae944e5c906e07d987177cc5) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.173763 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 4934bdde6023419fafd750920bcfe3d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 105b6148ae944e5c906e07d987177cc5) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:03.174140 847 leader_election.cc:343] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e3fbde011216426cb41c51fe359dcb5f
I20250115 14:52:03.174489 775 raft_consensus.cc:2463] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:03.174962 848 leader_election.cc:304] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:03.175432 1843 raft_consensus.cc:2798] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:03.175706 1843 raft_consensus.cc:695] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:03.177690 1164 tablet_bootstrap.cc:492] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:03.177860 1164 ts_tablet_manager.cc:1397] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20250115 14:52:03.178543 1164 raft_consensus.cc:357] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.178755 1164 raft_consensus.cc:383] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.178838 1164 raft_consensus.cc:738] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.168916 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 530) took 2172 ms. Trace:
I20250115 14:52:03.184741 1164 consensus_queue.cc:260] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.185920 1164 ts_tablet_manager.cc:1428] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:52:03.184823 1167 rpcz_store.cc:276] 0115 14:52:00.996698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.200709 (+204011us) service_pool.cc:225] Handling call
0115 14:52:03.168799 (+1968090us) ts_tablet_manager.cc:1145] Deleting tablet 0cae4c9228c640e495507e8f7ad9f544
0115 14:52:03.168808 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.168900 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1967913}
W20250115 14:52:03.186614 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0cae4c9228c640e495507e8f7ad9f544 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cae4c9228c640e495507e8f7ad9f544 already in progress: creating tablet
I20250115 14:52:03.186753 1164 tablet_bootstrap.cc:492] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.187644 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 531) took 2190 ms. Trace:
I20250115 14:52:03.189962 741 tablet_service.cc:1514] Processing DeleteTablet for tablet e7100e6a51ed4e178e86138f10d5ffc2 with delete_type TABLET_DATA_DELETED (replaced by tablet 6942d102069d4c33b23fa2c0b29f94a6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.190171 1167 rpcz_store.cc:276] 0115 14:52:00.996797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.200904 (+204107us) service_pool.cc:225] Handling call
0115 14:52:03.187517 (+1986613us) ts_tablet_manager.cc:1145] Deleting tablet 6308f68e41554470b7d1321c60d4eee9
0115 14:52:03.187528 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.187626 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1970612}
W20250115 14:52:03.190676 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6308f68e41554470b7d1321c60d4eee9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6308f68e41554470b7d1321c60d4eee9 already in progress: creating tablet
I20250115 14:52:03.190794 1751 ts_tablet_manager.cc:1905] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.190964 1162 tablet_bootstrap.cc:492] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.191113 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 532) took 2194 ms. Trace:
I20250115 14:52:03.191143 1162 ts_tablet_manager.cc:1397] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.003s
I20250115 14:52:03.173460 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 4934bdde6023419fafd750920bcfe3d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 105b6148ae944e5c906e07d987177cc5) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.191879 1162 raft_consensus.cc:357] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.192059 1162 raft_consensus.cc:383] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.192107 1162 raft_consensus.cc:738] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.192308 1162 consensus_queue.cc:260] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.194548 1164 tablet_bootstrap.cc:654] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.196220 1162 ts_tablet_manager.cc:1428] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:52:03.196513 1162 tablet_bootstrap.cc:492] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.199939 1162 tablet_bootstrap.cc:654] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:03.199947 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 338c0f6f39fd4335bbcda733b28b8bf8 consensus state has no leader
I20250115 14:52:03.200445 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e9e5bc6131944fcab9c550caac4f8d85
I20250115 14:52:03.201365 741 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e5bc6131944fcab9c550caac4f8d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.201570 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e924821d12045f6bc658461675aead4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.201751 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e5bc6131944fcab9c550caac4f8d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.202183 882 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e5bc6131944fcab9c550caac4f8d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.203888 1167 rpcz_store.cc:276] 0115 14:52:00.996872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.216924 (+220052us) service_pool.cc:225] Handling call
0115 14:52:03.190997 (+1974073us) ts_tablet_manager.cc:1145] Deleting tablet b5177265ba974ad6a381adb1e8c49589
0115 14:52:03.191007 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.191096 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1973913}
W20250115 14:52:03.204420 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b5177265ba974ad6a381adb1e8c49589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5177265ba974ad6a381adb1e8c49589 already in progress: creating tablet
I20250115 14:52:03.206672 1876 raft_consensus.cc:491] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.206789 1876 raft_consensus.cc:513] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.206995 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 537) took 1937 ms. Trace:
I20250115 14:52:03.207067 1167 rpcz_store.cc:276] 0115 14:52:01.267236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.267561 (+ 325us) service_pool.cc:225] Handling call
0115 14:52:03.204169 (+1936608us) ts_tablet_manager.cc:1145] Deleting tablet 0fd94e4a7fef44f5be462e7a77e8a123
0115 14:52:03.204179 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.204279 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1927871}
I20250115 14:52:03.207361 1876 leader_election.cc:290] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.207525 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 544) took 1710 ms. Trace:
I20250115 14:52:03.209091 1167 rpcz_store.cc:276] 0115 14:52:01.496956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.497944 (+ 988us) service_pool.cc:225] Handling call
0115 14:52:03.207289 (+1709345us) ts_tablet_manager.cc:1145] Deleting tablet 104282a09c024bbaafeb5f28f91dadcb
0115 14:52:03.207299 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.207508 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1709094}
I20250115 14:52:03.209604 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 546) took 1648 ms. Trace:
I20250115 14:52:03.210101 1167 rpcz_store.cc:276] 0115 14:52:01.561614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.562572 (+ 958us) service_pool.cc:225] Handling call
0115 14:52:03.209508 (+1646936us) ts_tablet_manager.cc:1145] Deleting tablet 496dec75500347699625a755dceed2bb
0115 14:52:03.209517 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.209589 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646742}
I20250115 14:52:03.210425 2058 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:45653 every 8 connection(s)
I20250115 14:52:03.210728 1168 tablet_bootstrap.cc:492] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.210850 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 547) took 1649 ms. Trace:
I20250115 14:52:03.211230 1808 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:45653
I20250115 14:52:03.210906 1168 ts_tablet_manager.cc:1397] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.010s
I20250115 14:52:03.211331 1167 rpcz_store.cc:276] 0115 14:52:01.561724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.563765 (+ 2041us) service_pool.cc:225] Handling call
0115 14:52:03.210755 (+1646990us) ts_tablet_manager.cc:1145] Deleting tablet a2fe51b51994491ab38a644bfc55e3ac
0115 14:52:03.210764 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.210836 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1619276}
I20250115 14:52:03.212103 1168 raft_consensus.cc:357] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.212917 1168 raft_consensus.cc:383] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.213025 1168 raft_consensus.cc:738] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.213289 1808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-1/data/info.pb
I20250115 14:52:03.213299 1168 consensus_queue.cc:260] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:03.214097 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0fd94e4a7fef44f5be462e7a77e8a123 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fd94e4a7fef44f5be462e7a77e8a123 already in progress: creating tablet
I20250115 14:52:03.214365 1168 ts_tablet_manager.cc:1428] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20250115 14:52:03.214359 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 104282a09c024bbaafeb5f28f91dadcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 104282a09c024bbaafeb5f28f91dadcb already in progress: creating tablet
W20250115 14:52:03.214552 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 496dec75500347699625a755dceed2bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 496dec75500347699625a755dceed2bb already in progress: creating tablet
I20250115 14:52:03.214612 1168 tablet_bootstrap.cc:492] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:03.214762 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 545) took 1656 ms. Trace:
W20250115 14:52:03.214774 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a2fe51b51994491ab38a644bfc55e3ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2fe51b51994491ab38a644bfc55e3ac already in progress: creating tablet
I20250115 14:52:03.214830 1167 rpcz_store.cc:276] 0115 14:52:01.558572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.558826 (+ 254us) service_pool.cc:225] Handling call
0115 14:52:03.214643 (+1655817us) ts_tablet_manager.cc:1145] Deleting tablet 3f541e916d264738842cd965c7a2a5ec
0115 14:52:03.214651 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.214745 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621735}
I20250115 14:52:03.215204 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa1827366a34f039fd12deb855f07b6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.215935 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa1827366a34f039fd12deb855f07b6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.216154 1168 tablet_bootstrap.cc:654] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:03.216109 983 leader_election.cc:343] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:03.216521 984 leader_election.cc:343] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.216617 984 leader_election.cc:304] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.217175 1876 raft_consensus.cc:2743] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:03.217424 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3f541e916d264738842cd965c7a2a5ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f541e916d264738842cd965c7a2a5ec already in progress: creating tablet
I20250115 14:52:03.218199 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task__Z8ae/build/asan/bin/kudu as pid 1808
I20250115 14:52:03.220834 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 550) took 1492 ms. Trace:
I20250115 14:52:03.220957 1167 rpcz_store.cc:276] 0115 14:52:01.728322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.728470 (+ 148us) service_pool.cc:225] Handling call
0115 14:52:03.220727 (+1492257us) ts_tablet_manager.cc:1145] Deleting tablet 010e6535b6bc44f391e6f2b6f1686532
0115 14:52:03.220736 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.220817 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492023}
I20250115 14:52:03.221269 447 master_service.cc:946] ReplaceTablet: received request to replace tablet d97c2f443ab040f9bc1924f45379165e from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:03.221377 1167 tablet_replica.cc:331] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:03.221539 1167 raft_consensus.cc:2238] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.223704 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_544d606cf9af4accb670d91d7d717948" } modify_external_catalogs: true
I20250115 14:52:03.223886 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_544d606cf9af4accb670d91d7d717948" } modify_external_catalogs: true
I20250115 14:52:03.221719 1167 raft_consensus.cc:2267] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.226275 1167 ts_tablet_manager.cc:1905] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.227119 1166 ts_tablet_manager.cc:1918] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.229429 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5da68f9f7cdb439daae3b29bc1e40438" } modify_external_catalogs: true
I20250115 14:52:03.229616 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5da68f9f7cdb439daae3b29bc1e40438" } modify_external_catalogs: true
I20250115 14:52:03.230800 1838 consensus_queue.cc:1035] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:03.239352 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f3280e8ba6db4446aa3e3bfe2836abfa
I20250115 14:52:03.240680 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet f3280e8ba6db4446aa3e3bfe2836abfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.240072 751 tablet_service.cc:1514] Processing DeleteTablet for tablet f3280e8ba6db4446aa3e3bfe2836abfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.240906 1166 log.cc:1198] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/6f07ef30cac2470d9b2f428338dd37a3
I20250115 14:52:03.241201 882 tablet_service.cc:1514] Processing DeleteTablet for tablet f3280e8ba6db4446aa3e3bfe2836abfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.241243 1955 raft_consensus.cc:491] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.241350 1955 raft_consensus.cc:513] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.241941 1955 leader_election.cc:290] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.244141 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92231509c754e8dbfc564813a76a123" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.244366 913 raft_consensus.cc:2463] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
W20250115 14:52:03.244949 710 leader_election.cc:343] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b92231509c754e8dbfc564813a76a123
I20250115 14:52:03.245134 710 leader_election.cc:304] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:03.245554 1955 raft_consensus.cc:2798] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.245626 1955 raft_consensus.cc:491] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.245682 1955 raft_consensus.cc:3054] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.248114 1955 raft_consensus.cc:513] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.248807 1955 leader_election.cc:290] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.249533 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92231509c754e8dbfc564813a76a123" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:03.249691 913 raft_consensus.cc:3054] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.253573 913 raft_consensus.cc:2463] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:03.254412 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3ef67fd49f0b4888a1f78b8e3b4558f0" } new_table_name: "default.Table_6f0b6c724c1c43e7adf538ad7369b7a1" modify_external_catalogs: true
W20250115 14:52:03.254419 710 leader_election.cc:343] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b92231509c754e8dbfc564813a76a123
I20250115 14:52:03.254655 710 leader_election.cc:304] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:03.254616 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_349a30fe26984a52913598e0f22c9295"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.255007 1955 raft_consensus.cc:2798] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:52:03.255148 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_349a30fe26984a52913598e0f22c9295 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.255173 1955 raft_consensus.cc:695] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:03.255378 1955 consensus_queue.cc:237] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.255909 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_24d51c1e7d8d4fdd83f72d6094af1d6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.256714 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5b777235fb9842079aa9b036b1efd4c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.257138 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b777235fb9842079aa9b036b1efd4c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:03.260807 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24d51c1e7d8d4fdd83f72d6094af1d6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:03.262765 847 consensus_peers.cc:487] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd4102b1f83e414d93a6a68315e36d16. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:03.263177 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 consensus state has no leader
I20250115 14:52:03.267983 1876 raft_consensus.cc:491] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.268123 1876 raft_consensus.cc:513] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.268771 1876 leader_election.cc:290] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.269352 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3d2ca7cd7c48a79ed1396fbbdc6866" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.269997 984 leader_election.cc:343] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.270483 1162 tablet_bootstrap.cc:492] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.270614 1162 ts_tablet_manager.cc:1397] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.004s
I20250115 14:52:03.271349 1162 raft_consensus.cc:357] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.271530 1162 raft_consensus.cc:383] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.271596 1162 raft_consensus.cc:738] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.271803 1162 consensus_queue.cc:260] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.272326 1162 ts_tablet_manager.cc:1428] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:03.272389 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3d2ca7cd7c48a79ed1396fbbdc6866" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.272588 1162 tablet_bootstrap.cc:492] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.275202 1166 ts_tablet_manager.cc:1939] T 6f07ef30cac2470d9b2f428338dd37a3 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.275897 1164 tablet_bootstrap.cc:492] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
W20250115 14:52:03.275933 983 leader_election.cc:343] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.276024 1164 ts_tablet_manager.cc:1397] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.013s
I20250115 14:52:03.276039 983 leader_election.cc:304] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.276125 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 6f07ef30cac2470d9b2f428338dd37a3 (table default.Table_17b896568dc34b1aaf2b3e197d03fdc4 [id=b9d32cae6ebd42cea68dc15f454318ab]) successfully deleted
I20250115 14:52:03.276214 1166 tablet_replica.cc:331] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.276403 1166 raft_consensus.cc:2238] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.276510 1876 raft_consensus.cc:2743] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.276695 1164 raft_consensus.cc:357] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.276885 1164 raft_consensus.cc:383] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.276926 1164 raft_consensus.cc:738] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.278126 1876 raft_consensus.cc:491] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.278232 1876 raft_consensus.cc:513] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.278497 1164 consensus_queue.cc:260] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.278733 1876 leader_election.cc:290] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.278959 1164 ts_tablet_manager.cc:1428] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:03.279196 1164 tablet_bootstrap.cc:492] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.279788 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d590c63d6e4d058fac50dc50010317" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.280663 984 leader_election.cc:343] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.281204 1164 tablet_bootstrap.cc:654] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.281222 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d590c63d6e4d058fac50dc50010317" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.282007 1843 consensus_queue.cc:237] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:03.282480 983 leader_election.cc:343] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.282577 983 leader_election.cc:304] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.283063 2043 raft_consensus.cc:2743] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.285516 1166 raft_consensus.cc:2267] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.286029 1166 ts_tablet_manager.cc:1905] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.288007 1162 tablet_bootstrap.cc:654] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.290360 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2a410c48ef764ba2ac337b8471f11ba1" } new_table_name: "default.Table_8652d6d9ec084553b5183aa372c2554a" modify_external_catalogs: true
I20250115 14:52:03.291743 1838 raft_consensus.cc:491] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.291867 1838 raft_consensus.cc:513] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.292467 1838 leader_election.cc:290] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.293656 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496dec75500347699625a755dceed2bb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.294238 847 leader_election.cc:343] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 496dec75500347699625a755dceed2bb
W20250115 14:52:03.294657 848 leader_election.cc:343] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.294765 848 leader_election.cc:304] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.295197 1838 raft_consensus.cc:2743] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:03.296124 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2484885df0da46f381d625a26e211c1e consensus state has no leader
I20250115 14:52:03.297151 1751 ts_tablet_manager.cc:1918] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.301304 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5a8b86b253c04399a44480d59eeddea5" } new_table_name: "default.Table_352831c0b6b04ca6825725b7825ecb12" modify_external_catalogs: true
I20250115 14:52:03.301573 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9591925055134e48ad3592198934e708"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.302068 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9591925055134e48ad3592198934e708 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:03.304068 847 consensus_peers.cc:487] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d97c2f443ab040f9bc1924f45379165e. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:03.308213 1167 ts_tablet_manager.cc:1918] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:03.311964 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0b954f79fdf24d45a41a28c878ddbad9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.312518 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b954f79fdf24d45a41a28c878ddbad9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.313012 451 master_service.cc:946] ReplaceTablet: received request to replace tablet d9ce65604c294021af93632ed073f627 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:03.304262 1876 raft_consensus.cc:491] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.313814 1876 raft_consensus.cc:513] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.314400 1876 leader_election.cc:290] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.314716 1751 log.cc:1198] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d71a1ecd900942e5ae788c9a28c70a7a
I20250115 14:52:03.314710 1166 ts_tablet_manager.cc:1918] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.314846 1166 log.cc:1198] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/75788f8eb36b446e927197f624fbfdf5
I20250115 14:52:03.315008 1167 log.cc:1198] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/fe696560ba6b4a76ad013d9a3486e398
I20250115 14:52:03.315102 1751 ts_tablet_manager.cc:1939] T d71a1ecd900942e5ae788c9a28c70a7a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:03.315102 1166 ts_tablet_manager.cc:1939] T 75788f8eb36b446e927197f624fbfdf5 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.315739 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 629873c00550408798b00489ece4ea5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.316330 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 75788f8eb36b446e927197f624fbfdf5 (table default.Table_875d7bd9588344008279ca89c1ad52be [id=7f3b54e6dae14efcb524f4f03c0a57b1]) successfully deleted
I20250115 14:52:03.316327 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2ee5dfdef14ec092a2f9d4bfd295a6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.316443 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d71a1ecd900942e5ae788c9a28c70a7a (table default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]) successfully deleted
I20250115 14:52:03.316800 1167 ts_tablet_manager.cc:1939] T fe696560ba6b4a76ad013d9a3486e398 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
W20250115 14:52:03.316800 984 leader_election.cc:343] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.316954 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_705745b616194094bb03915f2f651122" } modify_external_catalogs: true
I20250115 14:52:03.317101 1751 tablet_replica.cc:331] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:03.317101 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 551) took 1583 ms. Trace:
I20250115 14:52:03.317107 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_705745b616194094bb03915f2f651122" } modify_external_catalogs: true
I20250115 14:52:03.317276 1751 raft_consensus.cc:2238] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.317165 1167 rpcz_store.cc:276] 0115 14:52:01.734055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.734106 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:03.221336 (+1487230us) ts_tablet_manager.cc:1145] Deleting tablet fe696560ba6b4a76ad013d9a3486e398
0115 14:52:03.221344 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.230100 (+ 8756us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.308113 (+ 78013us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.317085 (+ 8972us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1487001}
I20250115 14:52:03.309018 1168 tablet_bootstrap.cc:492] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.316027 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2ee5dfdef14ec092a2f9d4bfd295a6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.317510 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet fe696560ba6b4a76ad013d9a3486e398 (table default.Table_498abe3411a94a3d8bbbf3c96571396c [id=5bccc8cec8fc49d9aed2bc209bdf53cb]) successfully deleted
I20250115 14:52:03.317399 1751 raft_consensus.cc:2267] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.317829 1168 ts_tablet_manager.cc:1397] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.013s
W20250115 14:52:03.317835 983 leader_election.cc:343] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.317924 983 leader_election.cc:304] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.318225 1876 raft_consensus.cc:2743] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.318528 1168 raft_consensus.cc:357] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.318701 1168 raft_consensus.cc:383] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.318763 1168 raft_consensus.cc:738] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.318970 1751 ts_tablet_manager.cc:1905] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.318946 1168 consensus_queue.cc:260] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.320567 1168 ts_tablet_manager.cc:1428] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:03.320772 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_edd5bd0ae3dd4bd994b4e746c1c14c9a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.320968 1166 tablet_replica.cc:331] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.321130 1166 raft_consensus.cc:2238] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.321267 1166 raft_consensus.cc:2267] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:03.321275 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edd5bd0ae3dd4bd994b4e746c1c14c9a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.323267 1168 tablet_bootstrap.cc:492] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:03.324368 1166 ts_tablet_manager.cc:1905] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.324931 1162 tablet_bootstrap.cc:492] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.325098 1162 ts_tablet_manager.cc:1397] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.004s
I20250115 14:52:03.325337 1838 consensus_queue.cc:1035] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:03.326169 1162 raft_consensus.cc:357] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:03.329290 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be8e12ccb8d14c1c9258dc3f81d1527d consensus state has no leader
I20250115 14:52:03.329418 1162 raft_consensus.cc:383] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.329526 1162 raft_consensus.cc:738] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.331071 1162 consensus_queue.cc:260] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.331679 1162 ts_tablet_manager.cc:1428] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:52:03.332024 1162 tablet_bootstrap.cc:492] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.331943 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2918791b551d4aecb80aee6a40351a45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.337139 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2918791b551d4aecb80aee6a40351a45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.339335 1164 tablet_bootstrap.cc:492] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:03.339471 1164 ts_tablet_manager.cc:1397] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.003s
I20250115 14:52:03.340157 1164 raft_consensus.cc:357] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.340378 1164 raft_consensus.cc:383] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.340428 1164 raft_consensus.cc:738] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.340632 1164 consensus_queue.cc:260] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.340731 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 553) took 1592 ms. Trace:
I20250115 14:52:03.340898 1167 rpcz_store.cc:276] 0115 14:52:01.748017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.748325 (+ 308us) service_pool.cc:225] Handling call
0115 14:52:03.340595 (+1592270us) ts_tablet_manager.cc:1145] Deleting tablet 95718d39a7824517a36c49c3b86fef30
0115 14:52:03.340607 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.340710 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583191}
I20250115 14:52:03.341226 1164 ts_tablet_manager.cc:1428] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20250115 14:52:03.341629 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 95718d39a7824517a36c49c3b86fef30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 95718d39a7824517a36c49c3b86fef30 already in progress: creating tablet
I20250115 14:52:03.342027 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 554) took 1590 ms. Trace:
I20250115 14:52:03.342833 1167 rpcz_store.cc:276] 0115 14:52:01.751553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.751660 (+ 107us) service_pool.cc:225] Handling call
0115 14:52:03.341913 (+1590253us) ts_tablet_manager.cc:1145] Deleting tablet 8442dc919d5f4d63b9b2ce7276cc0bc5
0115 14:52:03.341922 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.342009 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1590091}
I20250115 14:52:03.342782 1168 tablet_bootstrap.cc:654] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.343976 1164 tablet_bootstrap.cc:492] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:03.344076 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 already in progress: creating tablet
I20250115 14:52:03.345530 1164 tablet_bootstrap.cc:654] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.345989 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 552) took 1601 ms. Trace:
I20250115 14:52:03.346127 1167 rpcz_store.cc:276] 0115 14:52:01.744476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.751899 (+ 7423us) service_pool.cc:225] Handling call
0115 14:52:03.345883 (+1593984us) ts_tablet_manager.cc:1145] Deleting tablet a48227ec5aa846c3ac63bd29ed38be25
0115 14:52:03.345893 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.345974 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1593876}
W20250115 14:52:03.346565 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a48227ec5aa846c3ac63bd29ed38be25 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48227ec5aa846c3ac63bd29ed38be25 already in progress: creating tablet
I20250115 14:52:03.347000 775 raft_consensus.cc:3054] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.346863 1167 tablet_replica.cc:331] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:03.349056 1167 raft_consensus.cc:2238] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.349233 1162 tablet_bootstrap.cc:654] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.349247 1167 raft_consensus.cc:2267] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.354472 1166 ts_tablet_manager.cc:1918] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:03.355074 1166 log.cc:1198] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d97c2f443ab040f9bc1924f45379165e
I20250115 14:52:03.356091 1166 ts_tablet_manager.cc:1939] T d97c2f443ab040f9bc1924f45379165e P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.356987 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dc4acc7152c24a22a0e7ac7de1943df3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.357527 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc4acc7152c24a22a0e7ac7de1943df3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.357717 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3657238a564a4fec99af530ff70671af" } new_table_name: "default.Table_2ef4d39c78514fb795a21054fb20be91" modify_external_catalogs: true
I20250115 14:52:03.358801 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f7ef4548b4c2429ea9bfa1fea3d621fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.359238 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7ef4548b4c2429ea9bfa1fea3d621fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.363164 1166 tablet_replica.cc:331] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.363359 1166 raft_consensus.cc:2238] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.363372 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d97c2f443ab040f9bc1924f45379165e (table default.Table_121cfa7da22441d581d821791e428785 [id=470df20dd8bd47f4acfa2d40f2e034bd]) successfully deleted
I20250115 14:52:03.363497 1166 raft_consensus.cc:2267] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.366557 1166 ts_tablet_manager.cc:1905] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.369786 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:03.370545 1162 tablet_bootstrap.cc:492] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.370685 1162 ts_tablet_manager.cc:1397] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.005s
I20250115 14:52:03.371404 1162 raft_consensus.cc:357] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.371577 1162 raft_consensus.cc:383] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.371621 1162 raft_consensus.cc:738] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.371829 1162 consensus_queue.cc:260] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.372462 1162 ts_tablet_manager.cc:1428] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:03.373987 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 742792be08444792892bd9cd7336dda3 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:03.375980 1166 ts_tablet_manager.cc:1918] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.376094 1166 log.cc:1198] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d4c984613c7d45d5b7c513cd900c9558
I20250115 14:52:03.376425 1955 raft_consensus.cc:491] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.376578 1162 tablet_bootstrap.cc:492] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.376545 1955 raft_consensus.cc:513] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.377125 1955 leader_election.cc:290] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.377983 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f3a87bf8b6452d8afd8dfd836ee4e6" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.377974 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f3a87bf8b6452d8afd8dfd836ee4e6" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:03.378178 1047 raft_consensus.cc:2463] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:03.378181 913 raft_consensus.cc:2463] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:03.378233 1162 tablet_bootstrap.cc:654] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.380178 710 leader_election.cc:304] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:03.380688 1955 raft_consensus.cc:2798] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.380759 1955 raft_consensus.cc:491] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.380810 1955 raft_consensus.cc:3054] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.383227 1955 raft_consensus.cc:513] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.384271 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f3a87bf8b6452d8afd8dfd836ee4e6" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:03.384450 1047 raft_consensus.cc:3054] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.386891 1047 raft_consensus.cc:2463] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:03.387478 1955 leader_election.cc:290] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.388129 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f3a87bf8b6452d8afd8dfd836ee4e6" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:03.388350 913 raft_consensus.cc:3054] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.390841 913 raft_consensus.cc:2463] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:03.387739 710 leader_election.cc:304] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:03.391285 2059 sys_catalog.cc:263] Verifying existing consensus state
I20250115 14:52:03.391705 1955 raft_consensus.cc:2798] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:03.391855 1955 raft_consensus.cc:695] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:03.392150 1955 consensus_queue.cc:237] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.389039 1166 ts_tablet_manager.cc:1939] T d4c984613c7d45d5b7c513cd900c9558 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.393749 1166 tablet_replica.cc:331] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.393854 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d4c984613c7d45d5b7c513cd900c9558 (table default.Table_272f9650cb3142f781f1d4d7c4769dfe [id=2fb684a787e4420aa9218b369eba5cf2]) successfully deleted
I20250115 14:52:03.393923 1166 raft_consensus.cc:2238] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:03.394105 1166 pending_rounds.cc:62] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:03.394161 1166 pending_rounds.cc:69] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558352337051648 op_type: NO_OP noop_request { }
I20250115 14:52:03.394255 1166 raft_consensus.cc:2883] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:03.394338 1166 raft_consensus.cc:2267] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.399927 1166 ts_tablet_manager.cc:1905] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:03.406503 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8589265cc054407b68055bf5665057e consensus state has no leader
I20250115 14:52:03.406893 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_66c26e1f10144558b65efccffc30c61f" } new_table_name: "default.Table_38d8cb2a1e1644e989325bc288b09b5d" modify_external_catalogs: true
I20250115 14:52:03.407372 1166 ts_tablet_manager.cc:1918] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:03.407454 1166 log.cc:1198] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/276abefcab6a422aa6467f00864c01f4
I20250115 14:52:03.407765 1166 ts_tablet_manager.cc:1939] T 276abefcab6a422aa6467f00864c01f4 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.408128 1166 tablet_replica.cc:331] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.408772 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 276abefcab6a422aa6467f00864c01f4 (table default.Table_c3fec71fedac4b4c9864b18ea6af10b9 [id=382fc98d4c614a9fb79d50a9d630b36d]) successfully deleted
I20250115 14:52:03.410296 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_86abf42208c1422c8cede2bf48d37c3a" } modify_external_catalogs: true
I20250115 14:52:03.410471 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_86abf42208c1422c8cede2bf48d37c3a" } modify_external_catalogs: true
I20250115 14:52:03.415501 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f418504cc929429c936bfb16aa538464"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.416221 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f418504cc929429c936bfb16aa538464 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.425297 446 catalog_manager.cc:5526] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d7ecece2ab49e1be7079216517d13b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d2d7ecece2ab49e1be7079216517d13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:03.425987 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:03.430567 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:03.432597 1166 raft_consensus.cc:2238] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.432426 1751 ts_tablet_manager.cc:1918] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.432801 1166 raft_consensus.cc:2267] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.432817 1751 log.cc:1198] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/caa1827366a34f039fd12deb855f07b6
I20250115 14:52:03.433357 1166 ts_tablet_manager.cc:1905] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.435958 1751 ts_tablet_manager.cc:1939] T caa1827366a34f039fd12deb855f07b6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:03.438228 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6f3f8cae6244968927b864e7ce97565 consensus state has no leader
I20250115 14:52:03.439229 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet caa1827366a34f039fd12deb855f07b6 (table default.Table_a43e4255e5c3414aa47bff8bbe111d13 [id=aa88c2d281fe4856b6d6af91c9359f23]) successfully deleted
I20250115 14:52:03.439836 1751 tablet_replica.cc:331] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:03.440523 1751 raft_consensus.cc:2238] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.440944 1751 raft_consensus.cc:2267] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.444311 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7e3ec000089a47a2930dec382192cd6f
I20250115 14:52:03.445065 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e3ec000089a47a2930dec382192cd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.445752 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e3ec000089a47a2930dec382192cd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.447351 1751 ts_tablet_manager.cc:1905] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.448653 2059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: Bootstrap starting.
I20250115 14:52:03.449638 741 tablet_service.cc:1467] Processing CreateTablet for tablet 328fbbc9e15b42b1ac5398d4cfec1ede (DEFAULT_TABLE table=default.Table_7aef87d795794fd4ae0d9a8776e8cd79 [id=9770da26981341db949de0e933593013]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.450148 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328fbbc9e15b42b1ac5398d4cfec1ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.450949 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 328fbbc9e15b42b1ac5398d4cfec1ede (DEFAULT_TABLE table=default.Table_7aef87d795794fd4ae0d9a8776e8cd79 [id=9770da26981341db949de0e933593013]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.451284 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328fbbc9e15b42b1ac5398d4cfec1ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.453331 1166 ts_tablet_manager.cc:1918] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:03.453423 1166 log.cc:1198] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f8c81f47cd6e4a308cd62124db38d42f
I20250115 14:52:03.453737 1166 ts_tablet_manager.cc:1939] T f8c81f47cd6e4a308cd62124db38d42f P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.454535 1166 tablet_replica.cc:331] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.454705 1166 raft_consensus.cc:2238] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.454839 1166 raft_consensus.cc:2267] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.458001 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f8c81f47cd6e4a308cd62124db38d42f (table default.Table_d8d3e6a881ce475a8f1319238ab8c610 [id=b6de0dbbced44c83af5f98a9d1020cfc]) successfully deleted
W20250115 14:52:03.458334 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 108260us
I20250115 14:52:03.458400 2013 consensus_queue.cc:1035] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:03.458495 1843 raft_consensus.cc:491] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.458632 1843 raft_consensus.cc:513] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.459072 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e3ec000089a47a2930dec382192cd6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.459196 1843 leader_election.cc:290] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.459358 878 tablet_service.cc:1514] Processing DeleteTablet for tablet e198b4e1a65b4704bf2bb8f7d5987385 with delete_type TABLET_DATA_DELETED (replaced by tablet 65a6f83730f74d13823bd15d93538df7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.459388 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 496dec75500347699625a755dceed2bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.459692 883 tablet_service.cc:1467] Processing CreateTablet for tablet 328fbbc9e15b42b1ac5398d4cfec1ede (DEFAULT_TABLE table=default.Table_7aef87d795794fd4ae0d9a8776e8cd79 [id=9770da26981341db949de0e933593013]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.460059 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 328fbbc9e15b42b1ac5398d4cfec1ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.460585 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e198b4e1a65b4704bf2bb8f7d5987385" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.461153 848 leader_election.cc:343] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.465867 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
W20250115 14:52:03.466293 847 leader_election.cc:343] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e198b4e1a65b4704bf2bb8f7d5987385
I20250115 14:52:03.466426 847 leader_election.cc:304] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.466888 1843 raft_consensus.cc:2743] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.471678 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:03.473531 1167 ts_tablet_manager.cc:1905] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.474897 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d3d1424de57f42b89ddb3d6f1cbff283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.475297 1166 ts_tablet_manager.cc:1905] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:03.475473 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3d1424de57f42b89ddb3d6f1cbff283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.477401 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet caed629878594954842dfc6a40cc8b6d
I20250115 14:52:03.479027 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ae36f7ba541a43ffad08ed6fc6b10ec0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.479568 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae36f7ba541a43ffad08ed6fc6b10ec0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.480912 741 tablet_service.cc:1514] Processing DeleteTablet for tablet caed629878594954842dfc6a40cc8b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.482046 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1c51a5aaced84915a1dba864731460fe" } modify_external_catalogs: true
I20250115 14:52:03.482220 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1c51a5aaced84915a1dba864731460fe" } modify_external_catalogs: true
I20250115 14:52:03.483397 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c16ddd6216634793b8438ac0421e8cb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.483879 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c16ddd6216634793b8438ac0421e8cb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.484851 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet caed629878594954842dfc6a40cc8b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.485229 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_55c2a1e317e34c3cabd92d742e6f2d6b" } new_table_name: "default.Table_545d92cbf2bd4f6c9b698d45a7112f3f" modify_external_catalogs: true
I20250115 14:52:03.481678 883 tablet_service.cc:1514] Processing DeleteTablet for tablet caed629878594954842dfc6a40cc8b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.491019 741 tablet_service.cc:1467] Processing CreateTablet for tablet c23b26e4b7ff4c699d1c86adf6731c71 (DEFAULT_TABLE table=default.Table_169aa44347fe4b50b95e7c62b9604402 [id=c337f42224b34681a816e76f682f9bb5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:03.491194 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 140610us
I20250115 14:52:03.497995 1166 ts_tablet_manager.cc:1918] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.498314 883 tablet_service.cc:1467] Processing CreateTablet for tablet c23b26e4b7ff4c699d1c86adf6731c71 (DEFAULT_TABLE table=default.Table_169aa44347fe4b50b95e7c62b9604402 [id=c337f42224b34681a816e76f682f9bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.499616 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c23b26e4b7ff4c699d1c86adf6731c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.498275 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c23b26e4b7ff4c699d1c86adf6731c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.504606 1166 log.cc:1198] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/0de6f6f014074c3d8775cb5e4121ccb0
I20250115 14:52:03.505239 1166 ts_tablet_manager.cc:1939] T 0de6f6f014074c3d8775cb5e4121ccb0 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.506421 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 0de6f6f014074c3d8775cb5e4121ccb0 (table default.Table_d7378da7b662410d864c634dd5e34926 [id=00bc79eb4ee84715bf870d54ae76300a]) successfully deleted
W20250115 14:52:03.507476 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5b85d63974ab418eb1921dd605fbf2b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b85d63974ab418eb1921dd605fbf2b0 already in progress: creating tablet
W20250115 14:52:03.508230 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e244689cef6844e2935becd265eecad0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e244689cef6844e2935becd265eecad0 already in progress: creating tablet
I20250115 14:52:03.511332 1020 tablet_service.cc:1467] Processing CreateTablet for tablet c23b26e4b7ff4c699d1c86adf6731c71 (DEFAULT_TABLE table=default.Table_169aa44347fe4b50b95e7c62b9604402 [id=c337f42224b34681a816e76f682f9bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.511770 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c23b26e4b7ff4c699d1c86adf6731c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.515102 1166 tablet_replica.cc:331] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
W20250115 14:52:03.515192 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c574dba6b2a04915bdd871de4dff4df0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c574dba6b2a04915bdd871de4dff4df0 already in progress: creating tablet
W20250115 14:52:03.515439 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b76d1bbf90c6416ca60805c9d8053bd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b76d1bbf90c6416ca60805c9d8053bd4 already in progress: creating tablet
W20250115 14:52:03.515647 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6942d102069d4c33b23fa2c0b29f94a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6942d102069d4c33b23fa2c0b29f94a6 already in progress: creating tablet
W20250115 14:52:03.515832 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 72a97d102c174bc3a5e3a3c5762033b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a97d102c174bc3a5e3a3c5762033b3 already in progress: creating tablet
I20250115 14:52:03.515836 1166 raft_consensus.cc:2238] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:03.516037 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1b2d04209fd34918b83978325a5e1df8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2d04209fd34918b83978325a5e1df8 already in progress: creating tablet
I20250115 14:52:03.516151 1166 raft_consensus.cc:2267] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.516902 1166 ts_tablet_manager.cc:1905] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:03.517300 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2560d42c15c64b4c9c0cfe95706b4d9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2560d42c15c64b4c9c0cfe95706b4d9e already in progress: creating tablet
I20250115 14:52:03.520219 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 02ce16fea4f9469c9fc06fa24f6554be
I20250115 14:52:03.521026 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_76ced15792e04bd0890267d54f928393" } new_table_name: "default.Table_8545a98d01ad4896aefc8375c21bef6f" modify_external_catalogs: true
I20250115 14:52:03.522795 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ce16fea4f9469c9fc06fa24f6554be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.522992 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ce16fea4f9469c9fc06fa24f6554be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.528538 2041 tablet.cc:1721] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:03.529331 882 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3393ed28d242838d6f2f170184f7e8 (DEFAULT_TABLE table=default.Table_e847214582ef4c838ae73922b0c49ef9 [id=c64e70cd3ca64677bf45dc26fc2c6d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.529690 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3393ed28d242838d6f2f170184f7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.531522 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3393ed28d242838d6f2f170184f7e8 (DEFAULT_TABLE table=default.Table_e847214582ef4c838ae73922b0c49ef9 [id=c64e70cd3ca64677bf45dc26fc2c6d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.532073 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ce16fea4f9469c9fc06fa24f6554be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.533129 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3393ed28d242838d6f2f170184f7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.539392 1166 ts_tablet_manager.cc:1918] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.539501 1166 log.cc:1198] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e3fbde011216426cb41c51fe359dcb5f
I20250115 14:52:03.539798 1166 ts_tablet_manager.cc:1939] T e3fbde011216426cb41c51fe359dcb5f P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.541172 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e3fbde011216426cb41c51fe359dcb5f (table default.Table_1cd00e265b464348bd1df215a1a0d9e9 [id=3fe917c536db4535b79e95a59f9ea8f8]) successfully deleted
W20250115 14:52:03.545871 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5265d5eeac83481ca97801e3475886da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5265d5eeac83481ca97801e3475886da already in progress: creating tablet
I20250115 14:52:03.546948 1166 tablet_replica.cc:331] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
W20250115 14:52:03.548628 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e5a059e2f0654a4aafe9e8082b8fde7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5a059e2f0654a4aafe9e8082b8fde7a already in progress: creating tablet
I20250115 14:52:03.548743 1164 tablet_bootstrap.cc:492] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
W20250115 14:52:03.549474 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ba596cfb3ce64718a42a9083975edc8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba596cfb3ce64718a42a9083975edc8d already in progress: creating tablet
I20250115 14:52:03.549650 1164 ts_tablet_manager.cc:1397] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.206s user 0.018s sys 0.023s
I20250115 14:52:03.550393 752 tablet_service.cc:1467] Processing CreateTablet for tablet 8472cb9e45264e29b2c21891ff40a859 (DEFAULT_TABLE table=default.Table_345eac8bafcc47b59e881aa44718957e [id=7bab541233934311a6ad70b1f8b187b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.550459 1164 raft_consensus.cc:357] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.550815 1164 raft_consensus.cc:383] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.550908 1164 raft_consensus.cc:738] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.551010 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 8472cb9e45264e29b2c21891ff40a859 (DEFAULT_TABLE table=default.Table_345eac8bafcc47b59e881aa44718957e [id=7bab541233934311a6ad70b1f8b187b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.551198 1164 consensus_queue.cc:260] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.551369 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8472cb9e45264e29b2c21891ff40a859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.551784 1164 ts_tablet_manager.cc:1428] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:03.551775 882 tablet_service.cc:1467] Processing CreateTablet for tablet ca5cc55b50ee4953800f10efe0ffb546 (DEFAULT_TABLE table=default.Table_ab10a36d2dd14df0b5b7dde01478a485 [id=30c6c8d04ba34bc2ba1ef39fcdc5c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.552109 1164 tablet_bootstrap.cc:492] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.552347 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5cc55b50ee4953800f10efe0ffb546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.558745 883 tablet_service.cc:1467] Processing CreateTablet for tablet 8472cb9e45264e29b2c21891ff40a859 (DEFAULT_TABLE table=default.Table_345eac8bafcc47b59e881aa44718957e [id=7bab541233934311a6ad70b1f8b187b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.559165 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8472cb9e45264e29b2c21891ff40a859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.560552 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8472cb9e45264e29b2c21891ff40a859. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.561676 1751 ts_tablet_manager.cc:1918] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.562330 1164 tablet_bootstrap.cc:654] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.563207 1751 log.cc:1198] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/104282a09c024bbaafeb5f28f91dadcb
I20250115 14:52:03.565871 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_62b16563153946409ee7c59069520182" } new_table_name: "default.Table_f0d5f1bc94b941bf9f5d55369bdfe2ef" modify_external_catalogs: true
I20250115 14:52:03.566200 751 tablet_service.cc:1467] Processing CreateTablet for tablet ca5cc55b50ee4953800f10efe0ffb546 (DEFAULT_TABLE table=default.Table_ab10a36d2dd14df0b5b7dde01478a485 [id=30c6c8d04ba34bc2ba1ef39fcdc5c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.566562 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5cc55b50ee4953800f10efe0ffb546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.566536 1751 ts_tablet_manager.cc:1939] T 104282a09c024bbaafeb5f28f91dadcb P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:03.572592 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9b06f855e23e47cbb239075f101b585c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.573832 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b06f855e23e47cbb239075f101b585c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.575038 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 104282a09c024bbaafeb5f28f91dadcb (table default.Table_06a952903c7b4dcc8088b8c9ff5a2ac5 [id=09205f90c7fc476185a1d31a60ad5fa0]) successfully deleted
I20250115 14:52:03.576207 2079 tablet.cc:1721] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:03.576859 1164 tablet_bootstrap.cc:492] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:03.576946 1168 tablet_bootstrap.cc:492] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.576998 1164 ts_tablet_manager.cc:1397] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.006s
I20250115 14:52:03.577179 1168 ts_tablet_manager.cc:1397] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.254s user 0.003s sys 0.003s
I20250115 14:52:03.577862 1164 raft_consensus.cc:357] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.578059 1164 raft_consensus.cc:383] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.578121 1164 raft_consensus.cc:738] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.577910 1168 raft_consensus.cc:357] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.578281 1168 raft_consensus.cc:383] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.578365 1168 raft_consensus.cc:738] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.578604 1168 consensus_queue.cc:260] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:03.578763 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e5a059e2f0654a4aafe9e8082b8fde7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5a059e2f0654a4aafe9e8082b8fde7a already in progress: creating tablet
W20250115 14:52:03.579854 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ba596cfb3ce64718a42a9083975edc8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba596cfb3ce64718a42a9083975edc8d already in progress: creating tablet
I20250115 14:52:03.580418 1751 tablet_replica.cc:331] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:03.581444 2013 raft_consensus.cc:491] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.581408 1164 consensus_queue.cc:260] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.581393 1751 raft_consensus.cc:2238] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.581553 2013 raft_consensus.cc:513] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.582011 1164 ts_tablet_manager.cc:1428] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:03.582134 2013 leader_election.cc:290] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.582270 1164 tablet_bootstrap.cc:492] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.582708 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caed629878594954842dfc6a40cc8b6d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:03.582953 1047 raft_consensus.cc:2463] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:03.583457 847 leader_election.cc:304] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:03.583822 1838 raft_consensus.cc:2798] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.583952 1838 raft_consensus.cc:491] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.584090 1838 raft_consensus.cc:3054] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.584872 1876 raft_consensus.cc:491] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.585001 1876 raft_consensus.cc:513] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.585567 1876 leader_election.cc:290] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.585947 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caed629878594954842dfc6a40cc8b6d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.586668 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet be63bae96c834a8bbbb1174c027935a2
I20250115 14:52:03.588073 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68c168048649c5b91600eb4302b0ce" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.588232 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet be63bae96c834a8bbbb1174c027935a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.588424 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68c168048649c5b91600eb4302b0ce" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.588644 983 leader_election.cc:343] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.589134 878 tablet_service.cc:1514] Processing DeleteTablet for tablet be63bae96c834a8bbbb1174c027935a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.589090 1838 raft_consensus.cc:513] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:03.589951 984 leader_election.cc:343] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.590073 984 leader_election.cc:304] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.590449 1876 raft_consensus.cc:2743] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.590554 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caed629878594954842dfc6a40cc8b6d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:03.590701 1047 raft_consensus.cc:3054] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.591069 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caed629878594954842dfc6a40cc8b6d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:03.593135 1047 raft_consensus.cc:2463] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:03.593398 2080 tablet.cc:1721] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:03.593569 847 leader_election.cc:304] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:03.593811 2013 raft_consensus.cc:2798] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:03.593900 2013 raft_consensus.cc:695] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:03.594127 2013 consensus_queue.cc:237] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.595492 1167 ts_tablet_manager.cc:1918] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.597683 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3393ed28d242838d6f2f170184f7e8 (DEFAULT_TABLE table=default.Table_e847214582ef4c838ae73922b0c49ef9 [id=c64e70cd3ca64677bf45dc26fc2c6d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.598237 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c3393ed28d242838d6f2f170184f7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.594067 1166 raft_consensus.cc:2238] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.600339 1166 raft_consensus.cc:2267] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.597564 1164 tablet_bootstrap.cc:654] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.602720 741 tablet_service.cc:1514] Processing DeleteTablet for tablet be63bae96c834a8bbbb1174c027935a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.604657 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d9ce65604c294021af93632ed073f627
I20250115 14:52:03.605173 451 catalog_manager.cc:6596] ReplaceTablet: tablet d9ce65604c294021af93632ed073f627 deleted and replaced by tablet 339dd408dab7430c980058af3af04433
I20250115 14:52:03.605034 1751 raft_consensus.cc:2267] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.606006 741 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ce65604c294021af93632ed073f627 with delete_type TABLET_DATA_DELETED (replaced by tablet 339dd408dab7430c980058af3af04433) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.606228 1164 tablet_bootstrap.cc:492] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:03.606341 1164 ts_tablet_manager.cc:1397] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20250115 14:52:03.606554 451 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
I20250115 14:52:03.607025 1164 raft_consensus.cc:357] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.607193 1164 raft_consensus.cc:383] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.607239 1164 raft_consensus.cc:738] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.607332 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9e128687957b4b559bb77fd82039e563"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.607839 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e128687957b4b559bb77fd82039e563 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.610220 1751 ts_tablet_manager.cc:1905] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.611033 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c72e28daf2c44e519f2e934f811954e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.611513 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c72e28daf2c44e519f2e934f811954e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.611986 1164 consensus_queue.cc:260] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.612959 1164 ts_tablet_manager.cc:1428] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250115 14:52:03.613674 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ce65604c294021af93632ed073f627 with delete_type TABLET_DATA_DELETED (replaced by tablet 339dd408dab7430c980058af3af04433) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.615360 1164 tablet_bootstrap.cc:492] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.616801 752 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c385edf4414157845b971aaf366831 with delete_type TABLET_DATA_DELETED (replaced by tablet 84844b5f50af408ab7dda73f7421cd9d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.619038 1876 raft_consensus.cc:491] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.619168 1876 raft_consensus.cc:513] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.619791 1876 leader_election.cc:290] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.621220 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ddfe9dfe14ba2b067f907a9d13538" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.621755 984 leader_election.cc:343] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.632489 1019 tablet_service.cc:1467] Processing CreateTablet for tablet ca5cc55b50ee4953800f10efe0ffb546 (DEFAULT_TABLE table=default.Table_ab10a36d2dd14df0b5b7dde01478a485 [id=30c6c8d04ba34bc2ba1ef39fcdc5c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.632944 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca5cc55b50ee4953800f10efe0ffb546. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.637257 1164 tablet_bootstrap.cc:654] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.640205 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:03.643127 1166 ts_tablet_manager.cc:1905] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.644137 752 tablet_service.cc:1467] Processing CreateTablet for tablet a349cfaff6be4b85a0b2cf3eee238c26 (DEFAULT_TABLE table=default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3 [id=e9499f3a239b4bcbb1485c11a2f315c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.644984 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a349cfaff6be4b85a0b2cf3eee238c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.647619 1019 tablet_service.cc:1467] Processing CreateTablet for tablet a349cfaff6be4b85a0b2cf3eee238c26 (DEFAULT_TABLE table=default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3 [id=e9499f3a239b4bcbb1485c11a2f315c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.647992 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a349cfaff6be4b85a0b2cf3eee238c26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.648770 1166 ts_tablet_manager.cc:1918] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:03.648866 1166 log.cc:1198] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/a3a7a487a42949bd9641c38d6c1fcfd3
I20250115 14:52:03.649587 1166 ts_tablet_manager.cc:1939] T a3a7a487a42949bd9641c38d6c1fcfd3 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.650730 1166 tablet_replica.cc:331] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.650535 1751 ts_tablet_manager.cc:1918] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:03.650903 1166 raft_consensus.cc:2238] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:03.651114 1166 pending_rounds.cc:62] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:03.651167 1166 pending_rounds.cc:69] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558354456621056 op_type: NO_OP noop_request { }
I20250115 14:52:03.651268 1166 raft_consensus.cc:2883] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:03.651345 1166 raft_consensus.cc:2267] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.652062 1166 ts_tablet_manager.cc:1905] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.653918 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet a3a7a487a42949bd9641c38d6c1fcfd3 (table default.Table_8b9b25dd8a85434a83d2641ffbc5189e [id=738170c5571a4d958944ed432428e99d]) successfully deleted
I20250115 14:52:03.655984 741 tablet_service.cc:1467] Processing CreateTablet for tablet e619d4aabff245f4b46f5d03cb9a856d (DEFAULT_TABLE table=default.Table_9915ea9ed9b749c8957dd5fb5152b5e5 [id=b673e0812fab4d4dbad6a2bad757f9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.656332 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e619d4aabff245f4b46f5d03cb9a856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.656432 1016 tablet_service.cc:1467] Processing CreateTablet for tablet e619d4aabff245f4b46f5d03cb9a856d (DEFAULT_TABLE table=default.Table_9915ea9ed9b749c8957dd5fb5152b5e5 [id=b673e0812fab4d4dbad6a2bad757f9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.656745 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e619d4aabff245f4b46f5d03cb9a856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.658773 752 tablet_service.cc:1467] Processing CreateTablet for tablet 64994f0d3fdd43e2935786bf11544fb6 (DEFAULT_TABLE table=default.Table_9c676839fdd84942afad4ab037b135ad [id=3bfe4b687140404fac3637f078dc8d4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.659061 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64994f0d3fdd43e2935786bf11544fb6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:03.595692 848 leader_election.cc:343] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.604251 1838 leader_election.cc:290] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:03.662240 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 66550.1us
W20250115 14:52:03.662474 848 leader_election.cc:343] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.623353 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ddfe9dfe14ba2b067f907a9d13538" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.663585 879 tablet_service.cc:1467] Processing CreateTablet for tablet 64994f0d3fdd43e2935786bf11544fb6 (DEFAULT_TABLE table=default.Table_9c676839fdd84942afad4ab037b135ad [id=3bfe4b687140404fac3637f078dc8d4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.663656 889 tablet_service.cc:1467] Processing CreateTablet for tablet e619d4aabff245f4b46f5d03cb9a856d (DEFAULT_TABLE table=default.Table_9915ea9ed9b749c8957dd5fb5152b5e5 [id=b673e0812fab4d4dbad6a2bad757f9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.663916 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64994f0d3fdd43e2935786bf11544fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.664094 878 tablet_service.cc:1467] Processing CreateTablet for tablet a349cfaff6be4b85a0b2cf3eee238c26 (DEFAULT_TABLE table=default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3 [id=e9499f3a239b4bcbb1485c11a2f315c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.664378 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a349cfaff6be4b85a0b2cf3eee238c26. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:03.664429 983 leader_election.cc:343] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.664536 983 leader_election.cc:304] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.664734 1876 raft_consensus.cc:2743] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.665074 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e619d4aabff245f4b46f5d03cb9a856d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.665571 1162 tablet_bootstrap.cc:492] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.665678 1162 ts_tablet_manager.cc:1397] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.289s user 0.037s sys 0.021s
I20250115 14:52:03.666317 1162 raft_consensus.cc:357] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.666476 1162 raft_consensus.cc:383] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.666515 1162 raft_consensus.cc:738] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.666709 1162 consensus_queue.cc:260] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.669270 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:03.669534 1162 ts_tablet_manager.cc:1428] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:03.671030 1162 tablet_bootstrap.cc:492] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.644861 1168 ts_tablet_manager.cc:1428] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.068s user 0.001s sys 0.001s
I20250115 14:52:03.672159 1168 tablet_bootstrap.cc:492] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:03.674460 1162 tablet_bootstrap.cc:654] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.674603 1168 tablet_bootstrap.cc:654] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.676515 1166 ts_tablet_manager.cc:1918] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:03.676625 1166 log.cc:1198] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b92231509c754e8dbfc564813a76a123
I20250115 14:52:03.676914 1166 ts_tablet_manager.cc:1939] T b92231509c754e8dbfc564813a76a123 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.676939 882 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ce65604c294021af93632ed073f627 with delete_type TABLET_DATA_DELETED (replaced by tablet 339dd408dab7430c980058af3af04433) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.677117 1167 log.cc:1198] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b92231509c754e8dbfc564813a76a123
I20250115 14:52:03.677613 1167 ts_tablet_manager.cc:1939] T b92231509c754e8dbfc564813a76a123 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:03.680131 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 561) took 1829 ms. Trace:
I20250115 14:52:03.680253 1167 rpcz_store.cc:276] 0115 14:52:01.850415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.850695 (+ 280us) service_pool.cc:225] Handling call
0115 14:52:03.346819 (+1496124us) ts_tablet_manager.cc:1145] Deleting tablet b92231509c754e8dbfc564813a76a123
0115 14:52:03.346827 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.592239 (+245412us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.595406 (+ 3167us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.680113 (+ 84707us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1495938}
I20250115 14:52:03.680809 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b92231509c754e8dbfc564813a76a123 (table default.Table_02593da1cd7245f38f85424d1a4baad9 [id=1b38e621a58442c88f40bb9296563c36]) successfully deleted
I20250115 14:52:03.681003 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b92231509c754e8dbfc564813a76a123 (table default.Table_02593da1cd7245f38f85424d1a4baad9 [id=1b38e621a58442c88f40bb9296563c36]) successfully deleted
I20250115 14:52:03.681213 1167 tablet_replica.cc:331] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
W20250115 14:52:03.681607 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4934bdde6023419fafd750920bcfe3d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4934bdde6023419fafd750920bcfe3d4 already in progress: creating tablet
I20250115 14:52:03.682031 1166 tablet_replica.cc:331] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.682188 1166 raft_consensus.cc:2238] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.682332 1166 raft_consensus.cc:2267] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.682351 1167 raft_consensus.cc:2238] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.682569 1167 raft_consensus.cc:2267] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.682799 1166 ts_tablet_manager.cc:1905] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.683619 1167 ts_tablet_manager.cc:1905] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.689163 1164 tablet_bootstrap.cc:492] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:03.689296 1164 ts_tablet_manager.cc:1397] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.002s
I20250115 14:52:03.690002 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 994fe2bae335497f8c07f50b91377074 (DEFAULT_TABLE table=default.Table_33d4b270a59a4b438798f77331accb24 [id=ebef90c0843f48c2aacd0c2446d250a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.689993 1164 raft_consensus.cc:357] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.690157 1164 raft_consensus.cc:383] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.690204 1164 raft_consensus.cc:738] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.690389 1164 consensus_queue.cc:260] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.690514 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 994fe2bae335497f8c07f50b91377074. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.690865 1164 ts_tablet_manager.cc:1428] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:03.691084 1164 tablet_bootstrap.cc:492] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.693542 752 tablet_service.cc:1467] Processing CreateTablet for tablet 994fe2bae335497f8c07f50b91377074 (DEFAULT_TABLE table=default.Table_33d4b270a59a4b438798f77331accb24 [id=ebef90c0843f48c2aacd0c2446d250a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.696062 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 994fe2bae335497f8c07f50b91377074. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.701520 1164 tablet_bootstrap.cc:654] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.701835 882 tablet_service.cc:1467] Processing CreateTablet for tablet 994fe2bae335497f8c07f50b91377074 (DEFAULT_TABLE table=default.Table_33d4b270a59a4b438798f77331accb24 [id=ebef90c0843f48c2aacd0c2446d250a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.702247 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 994fe2bae335497f8c07f50b91377074. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.703107 1838 raft_consensus.cc:491] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.703223 1838 raft_consensus.cc:513] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:03.703612 847 consensus_peers.cc:487] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f07ef30cac2470d9b2f428338dd37a3. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:03.703873 1838 leader_election.cc:290] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:03.705111 847 leader_election.cc:343] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b2c385edf4414157845b971aaf366831
I20250115 14:52:03.704764 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c385edf4414157845b971aaf366831" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.708384 775 raft_consensus.cc:2463] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:03.709031 848 leader_election.cc:304] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:03.709367 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb
I20250115 14:52:03.709383 1838 raft_consensus.cc:2798] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.709446 1838 raft_consensus.cc:491] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.709499 1838 raft_consensus.cc:3054] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.711836 1838 raft_consensus.cc:513] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.712131 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.713191 752 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:03.714301 847 leader_election.cc:343] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b2c385edf4414157845b971aaf366831
I20250115 14:52:03.714499 752 tablet_service.cc:1467] Processing CreateTablet for tablet 5e10fe1445be457a8c848c30771af866 (DEFAULT_TABLE table=default.Table_9edd581e8fc4457697465360c4504ff1 [id=0ef6f07cf29e46e69bf2a4449187a4e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.714869 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e10fe1445be457a8c848c30771af866. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.714825 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c385edf4414157845b971aaf366831" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:03.714988 775 raft_consensus.cc:3054] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.715310 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 5e10fe1445be457a8c848c30771af866 (DEFAULT_TABLE table=default.Table_9edd581e8fc4457697465360c4504ff1 [id=0ef6f07cf29e46e69bf2a4449187a4e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.715621 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e10fe1445be457a8c848c30771af866. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.717464 775 raft_consensus.cc:2463] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:03.718025 848 leader_election.cc:304] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:03.718261 2013 raft_consensus.cc:2798] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:03.718364 2013 raft_consensus.cc:695] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:03.718560 1838 leader_election.cc:290] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.718586 2013 consensus_queue.cc:237] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.719682 889 tablet_service.cc:1467] Processing CreateTablet for tablet 5e10fe1445be457a8c848c30771af866 (DEFAULT_TABLE table=default.Table_9edd581e8fc4457697465360c4504ff1 [id=0ef6f07cf29e46e69bf2a4449187a4e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.720019 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e10fe1445be457a8c848c30771af866. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.720525 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_15364f05c206457da9c472858167e5dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.721082 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15364f05c206457da9c472858167e5dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:03.721297 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4655600323f944969cfc4eaf5bbbb684 consensus state has no leader
I20250115 14:52:03.721624 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a584af21c70545b8a2d3690c20422f1f" } new_table_name: "default.Table_ff33645127a443efb2b776d808942dba" modify_external_catalogs: true
W20250115 14:52:03.721100 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5e97279185541b992340c24ede01656 consensus state has no leader
I20250115 14:52:03.706283 1166 ts_tablet_manager.cc:1918] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.723333 1166 log.cc:1198] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e7100e6a51ed4e178e86138f10d5ffc2
I20250115 14:52:03.723595 1166 ts_tablet_manager.cc:1939] T e7100e6a51ed4e178e86138f10d5ffc2 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:52:03.724078 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6f2bc62621d402cacfcbe2eb74173f9 consensus state has no leader
I20250115 14:52:03.724889 1166 tablet_replica.cc:331] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.725075 1166 raft_consensus.cc:2238] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.725203 1166 raft_consensus.cc:2267] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.725879 1166 ts_tablet_manager.cc:1905] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.723129 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_683d27a9444245be9ff47c13b148e508"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.726989 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:03.727939 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_82a2ce9b462c48b28d9fa64f0f426ec8" } new_table_name: "default.Table_306d0c66a91b4c7784d68f7649a7dd69" modify_external_catalogs: true
I20250115 14:52:03.730342 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_79fecf866fd6405a9b7df222877fc2b0" } modify_external_catalogs: true
I20250115 14:52:03.730876 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_79fecf866fd6405a9b7df222877fc2b0" } modify_external_catalogs: true
W20250115 14:52:03.731935 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_683d27a9444245be9ff47c13b148e508 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.732311 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_087db70e94b0414c83610ae7e656c9d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.732641 1162 tablet_bootstrap.cc:492] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.732699 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e7100e6a51ed4e178e86138f10d5ffc2 (table default.Table_25a5a732ec814f24893d09a6f0b38d2c [id=fa886a7e8fcd43a096892c6435e13c29]) successfully deleted
I20250115 14:52:03.732787 1162 ts_tablet_manager.cc:1397] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.004s
W20250115 14:52:03.733160 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f3280e8ba6db4446aa3e3bfe2836abfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3280e8ba6db4446aa3e3bfe2836abfa already in progress: creating tablet
I20250115 14:52:03.733269 1838 raft_consensus.cc:491] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.733383 1838 raft_consensus.cc:513] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.733480 1162 raft_consensus.cc:357] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.733635 1162 raft_consensus.cc:383] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.733680 1162 raft_consensus.cc:738] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.733863 1162 consensus_queue.cc:260] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.733974 1838 leader_election.cc:290] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.734335 1162 ts_tablet_manager.cc:1428] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250115 14:52:03.734615 1162 tablet_bootstrap.cc:492] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.735368 752 tablet_service.cc:1514] Processing DeleteTablet for tablet bc8337d181d44ac7bb82551a677b6831 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:55 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.735841 1162 tablet_bootstrap.cc:654] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.736896 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13b4c0edc2c4fc9a1c527e1ff6a3d00" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:03.737087 1047 raft_consensus.cc:2463] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:03.737465 847 leader_election.cc:304] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:52:03.737660 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_087db70e94b0414c83610ae7e656c9d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.737903 2083 raft_consensus.cc:2798] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.737980 2083 raft_consensus.cc:491] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.738035 2083 raft_consensus.cc:3054] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.738941 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]). Expected version 1 got 0
I20250115 14:52:03.740085 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0763d4939e83417ab72a4b2fc39d2f33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:03.740427 1955 raft_consensus.cc:491] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:52:03.740576 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0763d4939e83417ab72a4b2fc39d2f33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.740547 1955 raft_consensus.cc:513] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.741112 1955 leader_election.cc:290] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.741863 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13b4c0edc2c4fc9a1c527e1ff6a3d00" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.742691 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caed629878594954842dfc6a40cc8b6d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:03.742921 1047 raft_consensus.cc:2388] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d7ecece2ab49e1be7079216517d13b in current term 1: Already voted for candidate 73c86158cc2641f182ea50a06634592d in this term.
I20250115 14:52:03.743542 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caed629878594954842dfc6a40cc8b6d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.744582 710 leader_election.cc:304] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:52:03.745002 848 leader_election.cc:343] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.745420 1955 raft_consensus.cc:3054] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.747741 1955 raft_consensus.cc:2743] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.747994 1838 consensus_queue.cc:1035] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:03.749779 1168 tablet_bootstrap.cc:492] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.749902 1168 ts_tablet_manager.cc:1397] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.000s
I20250115 14:52:03.750159 2083 raft_consensus.cc:513] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.750365 1838 raft_consensus.cc:491] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.750470 1838 raft_consensus.cc:513] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.750648 1168 raft_consensus.cc:357] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.750833 1168 raft_consensus.cc:383] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.750898 1168 raft_consensus.cc:738] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.751001 1838 leader_election.cc:290] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.751106 1168 consensus_queue.cc:260] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.751468 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13b4c0edc2c4fc9a1c527e1ff6a3d00" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:03.751633 1047 raft_consensus.cc:3054] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.751576 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 64994f0d3fdd43e2935786bf11544fb6 (DEFAULT_TABLE table=default.Table_9c676839fdd84942afad4ab037b135ad [id=3bfe4b687140404fac3637f078dc8d4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.753050 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64994f0d3fdd43e2935786bf11544fb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.753942 1047 raft_consensus.cc:2463] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:03.754789 847 leader_election.cc:304] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:03.755028 883 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.756031 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8337d181d44ac7bb82551a677b6831" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.756198 772 raft_consensus.cc:2463] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:03.756446 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13b4c0edc2c4fc9a1c527e1ff6a3d00" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:03.756973 848 leader_election.cc:304] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:03.750756 2083 leader_election.cc:290] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.760459 1838 raft_consensus.cc:2798] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:52:03.760495 847 leader_election.cc:343] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: bc8337d181d44ac7bb82551a677b6831
I20250115 14:52:03.760556 1838 raft_consensus.cc:695] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:03.760650 2084 raft_consensus.cc:491] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.760766 1838 consensus_queue.cc:237] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.760774 2084 raft_consensus.cc:513] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.761389 2084 leader_election.cc:290] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:03.761754 848 leader_election.cc:343] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.762102 752 tablet_service.cc:1467] Processing CreateTablet for tablet f4536bb3d44b40d2825e1f4e59d8a148 (DEFAULT_TABLE table=default.Table_892cf0050cb245a7af3e84a893239fba [id=92ab45b4bf154e789e3a9b33c77d38c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.762195 2013 raft_consensus.cc:2798] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.762317 2013 raft_consensus.cc:491] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.762445 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4536bb3d44b40d2825e1f4e59d8a148. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.762415 2013 raft_consensus.cc:3054] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.763981 2084 consensus_queue.cc:1035] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:03.768620 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffa997ca6c46559830b1dedf2d1a66" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.769454 848 leader_election.cc:343] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.772480 889 tablet_service.cc:1467] Processing CreateTablet for tablet f4536bb3d44b40d2825e1f4e59d8a148 (DEFAULT_TABLE table=default.Table_892cf0050cb245a7af3e84a893239fba [id=92ab45b4bf154e789e3a9b33c77d38c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.772486 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffa997ca6c46559830b1dedf2d1a66" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:03.772666 1047 raft_consensus.cc:2463] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:03.773052 847 leader_election.cc:304] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.773370 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:03.773562 2083 raft_consensus.cc:2798] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.773627 2083 raft_consensus.cc:491] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.773680 2083 raft_consensus.cc:3054] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.774055 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4536bb3d44b40d2825e1f4e59d8a148. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.775166 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
I20250115 14:52:03.775902 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4ec686b46a094818bd820c4f3854220b" } modify_external_catalogs: true
I20250115 14:52:03.775981 2083 raft_consensus.cc:513] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.776090 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4ec686b46a094818bd820c4f3854220b" } modify_external_catalogs: true
I20250115 14:52:03.779039 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 23cba5cc51d7414a9352caedcc362c9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.781342 752 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1a69b6de5d433f82579b1259548fd2 (DEFAULT_TABLE table=default.Table_dfd7391e00dc4815aedb1a4c64eba86f [id=1cdf416758254c679c3c785b101c65e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.781698 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:03.781742 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1a69b6de5d433f82579b1259548fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.782733 879 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1a69b6de5d433f82579b1259548fd2 (DEFAULT_TABLE table=default.Table_dfd7391e00dc4815aedb1a4c64eba86f [id=1cdf416758254c679c3c785b101c65e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.783044 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1a69b6de5d433f82579b1259548fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.784529 2013 raft_consensus.cc:513] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.785749 2013 leader_election.cc:290] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.785755 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8337d181d44ac7bb82551a677b6831" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:03.785934 775 raft_consensus.cc:3054] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.788561 775 raft_consensus.cc:2463] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:03.788971 848 leader_election.cc:304] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
W20250115 14:52:03.790463 847 leader_election.cc:343] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: bc8337d181d44ac7bb82551a677b6831
I20250115 14:52:03.791055 2083 leader_election.cc:290] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.791193 2083 raft_consensus.cc:2798] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:03.791275 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1ee3b83d4fee4651b3979b44d4903db6" } modify_external_catalogs: true
I20250115 14:52:03.791393 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffa997ca6c46559830b1dedf2d1a66" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:03.791481 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1ee3b83d4fee4651b3979b44d4903db6" } modify_external_catalogs: true
I20250115 14:52:03.791558 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffa997ca6c46559830b1dedf2d1a66" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:03.791735 1047 raft_consensus.cc:3054] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:03.791791 848 leader_election.cc:343] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.791996 2083 raft_consensus.cc:695] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:03.794075 1876 raft_consensus.cc:491] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.794174 1047 raft_consensus.cc:2463] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:03.794193 1876 raft_consensus.cc:513] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.794508 847 leader_election.cc:304] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.794734 1876 leader_election.cc:290] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.794857 2084 raft_consensus.cc:2798] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:03.794950 2084 raft_consensus.cc:695] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:03.792207 2083 consensus_queue.cc:237] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.795141 2084 consensus_queue.cc:237] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.795516 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f9262e808b4589a21257ed3fb0363b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:03.796001 983 leader_election.cc:343] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.796202 1016 tablet_service.cc:1467] Processing CreateTablet for tablet f4536bb3d44b40d2825e1f4e59d8a148 (DEFAULT_TABLE table=default.Table_892cf0050cb245a7af3e84a893239fba [id=92ab45b4bf154e789e3a9b33c77d38c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.796557 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f9262e808b4589a21257ed3fb0363b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.796717 741 tablet_service.cc:1467] Processing CreateTablet for tablet 336bf703400c4fadbe9a13986ce1fd86 (DEFAULT_TABLE table=default.Table_02969c37d40f489482e5eba20a9e2a8c [id=f29f6193992746e5bc5a0b5ee6a1a8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.797073 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 336bf703400c4fadbe9a13986ce1fd86. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:03.796974 984 leader_election.cc:343] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.801263 984 leader_election.cc:304] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.801434 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4536bb3d44b40d2825e1f4e59d8a148. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.799306 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1a69b6de5d433f82579b1259548fd2 (DEFAULT_TABLE table=default.Table_dfd7391e00dc4815aedb1a4c64eba86f [id=1cdf416758254c679c3c785b101c65e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.804594 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a1a69b6de5d433f82579b1259548fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.805395 879 tablet_service.cc:1467] Processing CreateTablet for tablet 336bf703400c4fadbe9a13986ce1fd86 (DEFAULT_TABLE table=default.Table_02969c37d40f489482e5eba20a9e2a8c [id=f29f6193992746e5bc5a0b5ee6a1a8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.801620 1876 raft_consensus.cc:2743] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.799105 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 336bf703400c4fadbe9a13986ce1fd86 (DEFAULT_TABLE table=default.Table_02969c37d40f489482e5eba20a9e2a8c [id=f29f6193992746e5bc5a0b5ee6a1a8d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.805723 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 336bf703400c4fadbe9a13986ce1fd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.805826 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 336bf703400c4fadbe9a13986ce1fd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.809890 1168 ts_tablet_manager.cc:1428] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.060s user 0.001s sys 0.001s
I20250115 14:52:03.810904 1168 tablet_bootstrap.cc:492] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:03.812453 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0b954f79fdf24d45a41a28c878ddbad9" } modify_external_catalogs: true
I20250115 14:52:03.812639 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0b954f79fdf24d45a41a28c878ddbad9" } modify_external_catalogs: true
I20250115 14:52:03.813596 1166 ts_tablet_manager.cc:1918] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.813709 1166 log.cc:1198] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/629873c00550408798b00489ece4ea5f
I20250115 14:52:03.813959 1166 ts_tablet_manager.cc:1939] T 629873c00550408798b00489ece4ea5f P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.814452 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 629873c00550408798b00489ece4ea5f (table default.Table_992f266d1ab24ee68d346afbfdec91eb [id=dfc41804229046b29a6d1230b41b9fa7]) successfully deleted
I20250115 14:52:03.814774 1168 tablet_bootstrap.cc:654] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.815327 741 tablet_service.cc:1467] Processing CreateTablet for tablet b36ca2df86b84a509fba08301f46a04f (DEFAULT_TABLE table=default.Table_803f17ac9c4e40ffb06dc6ac0faaf7ed [id=9c567acb55854e26a291ad147cd9f64a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.815716 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36ca2df86b84a509fba08301f46a04f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.817375 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b36ca2df86b84a509fba08301f46a04f (DEFAULT_TABLE table=default.Table_803f17ac9c4e40ffb06dc6ac0faaf7ed [id=9c567acb55854e26a291ad147cd9f64a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.817847 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36ca2df86b84a509fba08301f46a04f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:03.817497 889 tablet_service.cc:1467] Processing CreateTablet for tablet b36ca2df86b84a509fba08301f46a04f (DEFAULT_TABLE table=default.Table_803f17ac9c4e40ffb06dc6ac0faaf7ed [id=9c567acb55854e26a291ad147cd9f64a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:03.820822 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36ca2df86b84a509fba08301f46a04f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:03.821437 848 consensus_peers.cc:487] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f07ef30cac2470d9b2f428338dd37a3. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:03.822146 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba33eb3f5a40450db5cf115fbe8a17e7 consensus state has no leader
I20250115 14:52:03.823935 1167 ts_tablet_manager.cc:1918] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.827021 2080 tablet.cc:1721] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:03.828948 1751 log.cc:1198] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a3a7a487a42949bd9641c38d6c1fcfd3
I20250115 14:52:03.829361 1166 tablet_replica.cc:331] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.829533 1166 raft_consensus.cc:2238] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.829708 1166 raft_consensus.cc:2267] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.830603 1166 ts_tablet_manager.cc:1905] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.834903 1751 ts_tablet_manager.cc:1939] T a3a7a487a42949bd9641c38d6c1fcfd3 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:03.837810 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5df0250d08214a9783c392071126ed4a consensus state has no leader
I20250115 14:52:03.838433 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a3a7a487a42949bd9641c38d6c1fcfd3 (table default.Table_8b9b25dd8a85434a83d2641ffbc5189e [id=738170c5571a4d958944ed432428e99d]) successfully deleted
I20250115 14:52:03.839536 1751 tablet_replica.cc:331] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:03.841501 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 01291d41a2b345b4bfd00e6be0fb0129 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:03.845317 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]). Expected version 2 got 0
I20250115 14:52:03.847241 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b82924e735846b2914520684d1b8fc7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.854151 1751 raft_consensus.cc:2238] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.856012 1751 raft_consensus.cc:2267] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.857017 447 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]). Expected version 2 got 0
I20250115 14:52:03.827540 2079 tablet.cc:1721] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:03.835850 2013 raft_consensus.cc:491] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.860621 2013 raft_consensus.cc:513] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.862160 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff155a5c2b0479a90e74993de6f402d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:03.864521 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 4f89ec0a5d68408c897f43ab39df62b1 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:03.865134 2013 leader_election.cc:290] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.865523 1167 log.cc:1198] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/629873c00550408798b00489ece4ea5f
W20250115 14:52:03.865550 848 leader_election.cc:343] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.865908 1167 ts_tablet_manager.cc:1939] T 629873c00550408798b00489ece4ea5f P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:03.866957 2041 tablet.cc:1721] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:03.868009 1164 tablet_bootstrap.cc:492] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:03.868134 1164 ts_tablet_manager.cc:1397] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.177s user 0.017s sys 0.017s
I20250115 14:52:03.868157 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 562) took 2017 ms. Trace:
I20250115 14:52:03.868361 1167 rpcz_store.cc:276] 0115 14:52:01.850494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.850915 (+ 421us) service_pool.cc:225] Handling call
0115 14:52:03.681173 (+1830258us) ts_tablet_manager.cc:1145] Deleting tablet 629873c00550408798b00489ece4ea5f
0115 14:52:03.681182 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.686480 (+ 5298us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.823850 (+137370us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.868139 (+ 44289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1830124}
I20250115 14:52:03.869810 452 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 339dd408dab7430c980058af3af04433
I20250115 14:52:03.870393 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 629873c00550408798b00489ece4ea5f (table default.Table_992f266d1ab24ee68d346afbfdec91eb [id=dfc41804229046b29a6d1230b41b9fa7]) successfully deleted
I20250115 14:52:03.871129 1167 tablet_replica.cc:331] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:03.871347 1167 raft_consensus.cc:2238] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:03.871649 1167 pending_rounds.cc:62] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:03.871732 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]). Expected version 2 got 0
I20250115 14:52:03.871768 1167 pending_rounds.cc:69] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558350056185856 op_type: NO_OP noop_request { }
I20250115 14:52:03.871881 1167 raft_consensus.cc:2883] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:03.872345 1167 raft_consensus.cc:2267] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.872872 1167 ts_tablet_manager.cc:1905] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.873723 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet dff155a5c2b0479a90e74993de6f402d (table default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]). Expected version 1 got 0
I20250115 14:52:03.873935 452 catalog_manager.cc:5526] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:03.874331 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7]). Expected version 2 got 0
I20250115 14:52:03.866961 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff155a5c2b0479a90e74993de6f402d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:03.876129 1047 raft_consensus.cc:2463] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:03.876458 847 leader_election.cc:304] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.876916 1162 tablet_bootstrap.cc:492] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.876901 2083 raft_consensus.cc:2798] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.876968 2083 raft_consensus.cc:491] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.877038 2083 raft_consensus.cc:3054] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.877133 1162 ts_tablet_manager.cc:1397] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.143s user 0.006s sys 0.004s
I20250115 14:52:03.877784 1162 raft_consensus.cc:357] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.878021 1162 raft_consensus.cc:383] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.878106 1162 raft_consensus.cc:738] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.878351 1162 consensus_queue.cc:260] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.879312 2083 raft_consensus.cc:513] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.881013 2083 leader_election.cc:290] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.881304 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff155a5c2b0479a90e74993de6f402d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:03.881503 1047 raft_consensus.cc:3054] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.882997 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff155a5c2b0479a90e74993de6f402d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:03.883496 848 leader_election.cc:343] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.883642 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]). Expected version 2 got 0
I20250115 14:52:03.883978 1162 ts_tablet_manager.cc:1428] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:52:03.884131 1047 raft_consensus.cc:2463] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:03.884471 847 leader_election.cc:304] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.885545 1162 tablet_bootstrap.cc:492] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.885619 2013 raft_consensus.cc:2798] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:03.885730 2013 raft_consensus.cc:695] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:03.886121 2013 consensus_queue.cc:237] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.887308 1162 tablet_bootstrap.cc:654] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.893080 1164 raft_consensus.cc:357] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.893297 1164 raft_consensus.cc:383] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.893350 1164 raft_consensus.cc:738] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.896749 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9853dcdfee5f43de84798cfa0bef674e
I20250115 14:52:03.897013 1167 ts_tablet_manager.cc:1918] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:03.897111 1167 log.cc:1198] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/bd4102b1f83e414d93a6a68315e36d16
I20250115 14:52:03.897428 1167 ts_tablet_manager.cc:1939] T bd4102b1f83e414d93a6a68315e36d16 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:03.897974 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 9853dcdfee5f43de84798cfa0bef674e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.898422 1162 tablet_bootstrap.cc:492] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:03.898592 1162 ts_tablet_manager.cc:1397] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20250115 14:52:03.898833 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 9853dcdfee5f43de84798cfa0bef674e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.899286 1162 raft_consensus.cc:357] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.899505 1162 raft_consensus.cc:383] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.899600 1162 raft_consensus.cc:738] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.899938 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 563) took 2049 ms. Trace:
I20250115 14:52:03.899856 1162 consensus_queue.cc:260] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.900086 1167 rpcz_store.cc:276] 0115 14:52:01.850559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.851066 (+ 507us) service_pool.cc:225] Handling call
0115 14:52:03.870797 (+2019731us) ts_tablet_manager.cc:1145] Deleting tablet bd4102b1f83e414d93a6a68315e36d16
0115 14:52:03.871064 (+ 267us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.889854 (+ 18790us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.896931 (+ 7077us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.899922 (+ 2991us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019641}
I20250115 14:52:03.900602 1162 ts_tablet_manager.cc:1428] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250115 14:52:03.900609 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet bd4102b1f83e414d93a6a68315e36d16 (table default.Table_d0d86551c8474fb98ad74cb89ff84e28 [id=becd439cb45d48c3974b9ae72768542a]) successfully deleted
I20250115 14:52:03.900825 1751 ts_tablet_manager.cc:1905] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.900978 1167 tablet_replica.cc:331] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:03.902336 1164 consensus_queue.cc:260] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.902881 1164 ts_tablet_manager.cc:1428] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.035s user 0.000s sys 0.002s
I20250115 14:52:03.904878 1164 tablet_bootstrap.cc:492] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.909543 1167 raft_consensus.cc:2238] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.914224 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa
I20250115 14:52:03.914732 446 catalog_manager.cc:6596] ReplaceTablet: tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa deleted and replaced by tablet eb78eb2c5b9b44fba1a4e140a8e102d7
I20250115 14:52:03.901907 1168 tablet_bootstrap.cc:492] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.916432 1168 ts_tablet_manager.cc:1397] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.106s user 0.004s sys 0.004s
I20250115 14:52:03.916947 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa with delete_type TABLET_DATA_DELETED (replaced by tablet eb78eb2c5b9b44fba1a4e140a8e102d7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.917178 1168 raft_consensus.cc:357] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.917341 1168 raft_consensus.cc:383] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.917388 1168 raft_consensus.cc:738] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.917752 1168 consensus_queue.cc:260] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.918309 1168 ts_tablet_manager.cc:1428] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:03.918639 1168 tablet_bootstrap.cc:492] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:03.921015 1168 tablet_bootstrap.cc:654] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.921448 1166 ts_tablet_manager.cc:1918] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:03.901255 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 9853dcdfee5f43de84798cfa0bef674e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.915724 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa with delete_type TABLET_DATA_DELETED (replaced by tablet eb78eb2c5b9b44fba1a4e140a8e102d7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.922191 1166 log.cc:1198] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e9e5bc6131944fcab9c550caac4f8d85
I20250115 14:52:03.922611 1167 raft_consensus.cc:2267] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.922845 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 2e4c2707915e4e97ad6c66cb249c6d9e from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:03.923372 1167 ts_tablet_manager.cc:1905] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.923417 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0218bb8722464e328f7582eacf088585"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.924019 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0218bb8722464e328f7582eacf088585 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.926003 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_494dfb6d6b814d458aaa51b8e77e570f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.926508 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_494dfb6d6b814d458aaa51b8e77e570f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.928184 1162 tablet_bootstrap.cc:492] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:03.929967 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbcbd9770e46481a894a5352459d371d" } modify_external_catalogs: true
I20250115 14:52:03.930130 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbcbd9770e46481a894a5352459d371d" } modify_external_catalogs: true
I20250115 14:52:03.931636 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_98190516662f4514bd887e05fe683703"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:03.932152 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98190516662f4514bd887e05fe683703 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:03.932276 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:03.934139 1168 tablet_bootstrap.cc:492] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.934298 1168 ts_tablet_manager.cc:1397] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250115 14:52:03.934352 1162 tablet_bootstrap.cc:654] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.933846 1166 ts_tablet_manager.cc:1939] T e9e5bc6131944fcab9c550caac4f8d85 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:03.935003 1168 raft_consensus.cc:357] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.935191 1168 raft_consensus.cc:383] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.935252 1168 raft_consensus.cc:738] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.935775 1168 consensus_queue.cc:260] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.936342 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa with delete_type TABLET_DATA_DELETED (replaced by tablet eb78eb2c5b9b44fba1a4e140a8e102d7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.936481 1168 ts_tablet_manager.cc:1428] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:03.936762 1168 tablet_bootstrap.cc:492] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:03.937529 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 342bf8d4dc4244f982b1b63aaae0fb45
I20250115 14:52:03.938217 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_efb7e019ddea47a8954081c1715654e3" } new_table_name: "default.Table_8a0954ff11744933ba4752116830cdd7" modify_external_catalogs: true
I20250115 14:52:03.938365 1168 tablet_bootstrap.cc:654] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.939635 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 342bf8d4dc4244f982b1b63aaae0fb45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:03.940347 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 342bf8d4dc4244f982b1b63aaae0fb45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:03.942080 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 342bf8d4dc4244f982b1b63aaae0fb45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:03.942203 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e9e5bc6131944fcab9c550caac4f8d85 (table default.Table_609197916c9b4c81aebdaf70e78cc436 [id=33e3df28f2d445c5ba4e8b0ce4458c1a]) successfully deleted
I20250115 14:52:03.943925 1167 ts_tablet_manager.cc:1918] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.943521 1166 tablet_replica.cc:331] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:03.946036 1164 tablet_bootstrap.cc:654] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.949488 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet c0a621e1a08742aca3cd9f37b2da5774
I20250115 14:52:03.949792 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5b777235fb9842079aa9b036b1efd4c9" } new_table_name: "default.Table_e541499a248445d7929d628f87f2d65e" modify_external_catalogs: true
I20250115 14:52:03.950598 1166 raft_consensus.cc:2238] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.951059 1751 ts_tablet_manager.cc:1918] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:03.951712 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9af73c3d469c4f2099396a0d51abc8ea" } new_table_name: "default.Table_a99495ed1c544a6f825fa19b1e44009f" modify_external_catalogs: true
I20250115 14:52:03.954389 1751 log.cc:1198] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/fc3d2ca7cd7c48a79ed1396fbbdc6866
I20250115 14:52:03.957049 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed] alter complete (version 1)
I20250115 14:52:03.957125 1751 ts_tablet_manager.cc:1939] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:03.958055 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 (table default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]) successfully deleted
I20250115 14:52:03.955219 1166 raft_consensus.cc:2267] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.960371 1167 log.cc:1198] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e932d5b820f94130967eccce9e1542fa
I20250115 14:52:03.960646 1166 ts_tablet_manager.cc:1905] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:03.960805 1167 ts_tablet_manager.cc:1939] T e932d5b820f94130967eccce9e1542fa P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:03.961145 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 564) took 2110 ms. Trace:
I20250115 14:52:03.961324 1167 rpcz_store.cc:276] 0115 14:52:01.850650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.851169 (+ 519us) service_pool.cc:225] Handling call
0115 14:52:03.900929 (+2049760us) ts_tablet_manager.cc:1145] Deleting tablet e932d5b820f94130967eccce9e1542fa
0115 14:52:03.900940 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.940984 (+ 40044us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.943843 (+ 2859us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:03.961128 (+ 17285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2049626}
I20250115 14:52:03.961990 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 565) took 2090 ms. Trace:
I20250115 14:52:03.962123 1167 rpcz_store.cc:276] 0115 14:52:01.871833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.872135 (+ 302us) service_pool.cc:225] Handling call
0115 14:52:03.961872 (+2089737us) ts_tablet_manager.cc:1145] Deleting tablet 59d8fd4b26294e8ca6745b95f2e03231
0115 14:52:03.961881 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:03.961973 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2089543}
I20250115 14:52:03.962581 1167 tablet_replica.cc:331] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:03.962821 1167 raft_consensus.cc:2238] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.964589 1168 tablet_bootstrap.cc:492] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.964763 1168 ts_tablet_manager.cc:1397] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250115 14:52:03.965099 1164 tablet_bootstrap.cc:492] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:03.965224 1164 ts_tablet_manager.cc:1397] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.003s
I20250115 14:52:03.965619 1168 raft_consensus.cc:357] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.965850 1168 raft_consensus.cc:383] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.965943 1168 raft_consensus.cc:738] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.966183 1168 consensus_queue.cc:260] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.966840 1164 raft_consensus.cc:357] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.967036 1164 raft_consensus.cc:383] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.967078 1164 raft_consensus.cc:738] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.967274 1164 consensus_queue.cc:260] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.967856 1164 ts_tablet_manager.cc:1428] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:03.968221 1164 tablet_bootstrap.cc:492] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.971253 1164 tablet_bootstrap.cc:654] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.974224 2013 raft_consensus.cc:491] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.974364 2013 raft_consensus.cc:513] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.974999 2013 leader_election.cc:290] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:03.975843 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d195be2d543f4a8693045a024912c43a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.976667 848 leader_election.cc:343] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.976589 1751 tablet_replica.cc:331] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:03.977327 847 leader_election.cc:343] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d195be2d543f4a8693045a024912c43a
I20250115 14:52:03.977437 847 leader_election.cc:304] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.977910 1168 ts_tablet_manager.cc:1428] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250115 14:52:03.977571 1751 raft_consensus.cc:2238] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:03.978078 2013 raft_consensus.cc:2743] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.978521 1168 tablet_bootstrap.cc:492] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:03.978533 1751 raft_consensus.cc:2267] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:03.978907 1164 tablet_bootstrap.cc:492] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:03.979040 1164 ts_tablet_manager.cc:1397] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20250115 14:52:03.979707 1164 raft_consensus.cc:357] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.979897 1168 tablet_bootstrap.cc:654] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:03.979894 1164 raft_consensus.cc:383] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:03.979972 1164 raft_consensus.cc:738] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:03.980357 1955 raft_consensus.cc:491] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.980489 1955 raft_consensus.cc:513] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.981025 1955 leader_election.cc:290] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.981745 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cba5cc51d7414a9352caedcc362c9c" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:03.981909 913 raft_consensus.cc:2463] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
W20250115 14:52:03.982060 710 leader_election.cc:343] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 23cba5cc51d7414a9352caedcc362c9c
I20250115 14:52:03.980159 1164 consensus_queue.cc:260] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.982946 710 leader_election.cc:304] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:03.983160 1876 raft_consensus.cc:491] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:03.983217 1955 raft_consensus.cc:2798] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:03.983306 1955 raft_consensus.cc:491] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:03.983273 1876 raft_consensus.cc:513] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:03.983392 1955 raft_consensus.cc:3054] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:03.983832 1876 leader_election.cc:290] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.984418 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e5d44b8064abbbf1b6419a1eca1ed" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:03.984967 984 leader_election.cc:343] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.986006 1955 raft_consensus.cc:513] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:03.986536 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e5d44b8064abbbf1b6419a1eca1ed" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:03.987001 983 leader_election.cc:343] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:03.987102 983 leader_election.cc:304] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:03.987146 1955 leader_election.cc:290] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:03.987460 1876 raft_consensus.cc:2743] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:03.987614 1164 ts_tablet_manager.cc:1428] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:52:03.988078 1164 tablet_bootstrap.cc:492] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:03.989976 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet e932d5b820f94130967eccce9e1542fa (table default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]) successfully deleted
W20250115 14:52:03.990347 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 4934bdde6023419fafd750920bcfe3d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4934bdde6023419fafd750920bcfe3d4 already in progress: creating tablet
I20250115 14:52:03.990583 1164 tablet_bootstrap.cc:654] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:03.992662 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 59d8fd4b26294e8ca6745b95f2e03231 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59d8fd4b26294e8ca6745b95f2e03231 already in progress: creating tablet
I20250115 14:52:03.998384 1168 tablet_bootstrap.cc:492] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:03.998574 1168 ts_tablet_manager.cc:1397] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20250115 14:52:03.999328 1168 raft_consensus.cc:357] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.000352 1168 raft_consensus.cc:383] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.000516 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fa4c201ba84d4cf5902269c4c6e4f50c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.000680 1168 raft_consensus.cc:738] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.000927 1168 consensus_queue.cc:260] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:04.001067 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa4c201ba84d4cf5902269c4c6e4f50c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.001910 1751 ts_tablet_manager.cc:1905] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.003614 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8be6e58a536c49c5add0072a13fa8be5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.003947 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 655489ad9bd34be0a8283a295f583092
W20250115 14:52:04.004117 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8be6e58a536c49c5add0072a13fa8be5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.004698 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_94471b2570804dbca5f0b06961b996e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.005143 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94471b2570804dbca5f0b06961b996e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.005735 1168 ts_tablet_manager.cc:1428] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:52:04.009109 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 655489ad9bd34be0a8283a295f583092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.009395 1168 tablet_bootstrap.cc:492] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:04.010766 1168 tablet_bootstrap.cc:654] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.011641 1162 tablet_bootstrap.cc:492] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:04.011832 1162 ts_tablet_manager.cc:1397] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.000s
I20250115 14:52:04.012605 1162 raft_consensus.cc:357] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.012841 1162 raft_consensus.cc:383] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.012939 1162 raft_consensus.cc:738] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.013260 1162 consensus_queue.cc:260] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.013646 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 655489ad9bd34be0a8283a295f583092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.013785 1162 ts_tablet_manager.cc:1428] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250115 14:52:04.018631 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65a6f83730f74d13823bd15d93538df7 consensus state has no leader
I20250115 14:52:04.018898 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 4821224ac8f1472d8b1e8afc06a118a3 from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:04.020100 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f924700fa524bae96016b039ab3b819 has no consensus state
I20250115 14:52:04.020437 1162 tablet_bootstrap.cc:492] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:04.020537 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74d00bcee22d40769102eacaf728480d consensus state has no leader
I20250115 14:52:04.022053 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4ced70cc9c3d4302b3856c7dc634d096"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.022557 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ced70cc9c3d4302b3856c7dc634d096 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.021770 1162 tablet_bootstrap.cc:654] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.027170 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23cba5cc51d7414a9352caedcc362c9c" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:04.027367 913 raft_consensus.cc:3054] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.027371 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 655489ad9bd34be0a8283a295f583092 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:04.027650 710 leader_election.cc:343] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 23cba5cc51d7414a9352caedcc362c9c
I20250115 14:52:04.030020 913 raft_consensus.cc:2463] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:04.030478 710 leader_election.cc:304] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:04.031715 1167 raft_consensus.cc:2267] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.035362 1167 ts_tablet_manager.cc:1905] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.039384 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_545d92cbf2bd4f6c9b698d45a7112f3f" } new_table_name: "default.Table_135f5999ba2b42fab38c7442ee051e44" modify_external_catalogs: true
I20250115 14:52:04.039937 2013 raft_consensus.cc:491] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.040063 2013 raft_consensus.cc:513] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.040593 2013 leader_election.cc:290] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:04.042265 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a602da76783c41a89e586a035929a1ba" } new_table_name: "default.Table_f9271e15235f4b92b6f54d3decfb8a4e" modify_external_catalogs: true
W20250115 14:52:04.044325 847 leader_election.cc:343] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 59d8fd4b26294e8ca6745b95f2e03231
I20250115 14:52:04.045142 1166 ts_tablet_manager.cc:1918] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:04.045254 1166 log.cc:1198] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/7e3ec000089a47a2930dec382192cd6f
I20250115 14:52:04.045560 1166 ts_tablet_manager.cc:1939] T 7e3ec000089a47a2930dec382192cd6f P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:04.046460 1166 tablet_replica.cc:331] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:04.046644 1166 raft_consensus.cc:2238] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.046778 1166 raft_consensus.cc:2267] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.048507 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4821224ac8f1472d8b1e8afc06a118a3
I20250115 14:52:04.049046 451 catalog_manager.cc:6596] ReplaceTablet: tablet 4821224ac8f1472d8b1e8afc06a118a3 deleted and replaced by tablet 63f57c1af0b541d992e6f0f9b02fc554
I20250115 14:52:04.050034 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 4821224ac8f1472d8b1e8afc06a118a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 63f57c1af0b541d992e6f0f9b02fc554) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.050606 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4821224ac8f1472d8b1e8afc06a118a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 63f57c1af0b541d992e6f0f9b02fc554) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:04.060261 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09adb34bf9a3464798b9f28c50d6fc11 consensus state has no leader
W20250115 14:52:04.060563 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5e97279185541b992340c24ede01656 consensus state has no leader
W20250115 14:52:04.074764 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 74d00bcee22d40769102eacaf728480d (table default.Table_a99495ed1c544a6f825fa19b1e44009f [id=1ccab9d1b41c4795b17a4bf435ece517], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 74d00bcee22d40769102eacaf728480d consensus state has no leader
I20250115 14:52:04.075778 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2e4c2707915e4e97ad6c66cb249c6d9e
I20250115 14:52:04.076258 446 catalog_manager.cc:6596] ReplaceTablet: tablet 2e4c2707915e4e97ad6c66cb249c6d9e deleted and replaced by tablet 6afbdd0b585e4e7e82067dce061924d0
I20250115 14:52:04.076872 1166 ts_tablet_manager.cc:1905] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.077347 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4c2707915e4e97ad6c66cb249c6d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6afbdd0b585e4e7e82067dce061924d0) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.077561 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4c2707915e4e97ad6c66cb249c6d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6afbdd0b585e4e7e82067dce061924d0) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.081357 1162 tablet_bootstrap.cc:492] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:04.081488 1162 ts_tablet_manager.cc:1397] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.001s
I20250115 14:52:04.081542 448 catalog_manager.cc:5526] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:04.081971 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet dff155a5c2b0479a90e74993de6f402d (table default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]). Expected version 1 got 0
I20250115 14:52:04.082162 1162 raft_consensus.cc:357] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.082259 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_7b4753acbcca4cdc94831798b3d47baa [id=7628dbb6818f443680a97fef665d6d2b]). Expected version 2 got 0
I20250115 14:52:04.082326 1162 raft_consensus.cc:383] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.082378 1162 raft_consensus.cc:738] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.082468 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]). Expected version 2 got 0
I20250115 14:52:04.082607 1162 consensus_queue.cc:260] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.083174 448 catalog_manager.cc:5526] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:04.083566 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
I20250115 14:52:04.086251 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_494dfb6d6b814d458aaa51b8e77e570f" } modify_external_catalogs: true
I20250115 14:52:04.086429 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_494dfb6d6b814d458aaa51b8e77e570f" } modify_external_catalogs: true
I20250115 14:52:04.087604 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:04.089188 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4c30bccb3549e6963dfbbecf68cbbe (DEFAULT_TABLE table=default.Table_9e128687957b4b559bb77fd82039e563 [id=856ea3ceeee84f3283d50a003e582724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.089560 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d4c30bccb3549e6963dfbbecf68cbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.093343 1162 ts_tablet_manager.cc:1428] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250115 14:52:04.096913 1751 ts_tablet_manager.cc:1918] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.098167 1751 log.cc:1198] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/0e924821d12045f6bc658461675aead4
I20250115 14:52:04.098767 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4c30bccb3549e6963dfbbecf68cbbe (DEFAULT_TABLE table=default.Table_9e128687957b4b559bb77fd82039e563 [id=856ea3ceeee84f3283d50a003e582724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.099261 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d4c30bccb3549e6963dfbbecf68cbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.100912 889 tablet_service.cc:1467] Processing CreateTablet for tablet e9febafb7e354e289cd3f864311f14cb (DEFAULT_TABLE table=default.Table_c72e28daf2c44e519f2e934f811954e1 [id=9fd8139f0247410d8aad7dcb63901905]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.101281 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9febafb7e354e289cd3f864311f14cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.102746 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1bef4c655e464405a4b2893dba5c9eca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.102805 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_05f540a8c52d40b1b3240deb036757ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.103312 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bef4c655e464405a4b2893dba5c9eca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.104395 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_721928c380b840ad84f9f5030b52632f" } modify_external_catalogs: true
I20250115 14:52:04.104511 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_67307f5008004a1fa8f71bf061edd60a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.104862 1162 tablet_bootstrap.cc:492] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:04.104943 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67307f5008004a1fa8f71bf061edd60a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.104590 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_721928c380b840ad84f9f5030b52632f" } modify_external_catalogs: true
W20250115 14:52:04.103324 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05f540a8c52d40b1b3240deb036757ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.105968 451 master_service.cc:946] ReplaceTablet: received request to replace tablet e180e37a1d5d46d78380323662b80f85 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:04.106406 1162 tablet_bootstrap.cc:654] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.106999 1167 ts_tablet_manager.cc:1918] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.107093 1167 log.cc:1198] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e7100e6a51ed4e178e86138f10d5ffc2
I20250115 14:52:04.107369 1167 ts_tablet_manager.cc:1939] T e7100e6a51ed4e178e86138f10d5ffc2 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:04.107632 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 566) took 2182 ms. Trace:
I20250115 14:52:04.107699 1167 rpcz_store.cc:276] 0115 14:52:01.925560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.925605 (+ 45us) service_pool.cc:225] Handling call
0115 14:52:03.962539 (+2036934us) ts_tablet_manager.cc:1145] Deleting tablet e7100e6a51ed4e178e86138f10d5ffc2
0115 14:52:03.962548 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.073420 (+110872us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.106907 (+ 33487us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.107617 (+ 710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2036769}
I20250115 14:52:04.108006 1167 tablet_replica.cc:331] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:04.108168 1167 raft_consensus.cc:2238] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.108332 1167 raft_consensus.cc:2267] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.108829 1167 ts_tablet_manager.cc:1905] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.114661 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 83e4f2ad27514868a224c543d4a28176
I20250115 14:52:04.116111 1168 tablet_bootstrap.cc:492] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:04.116237 1168 ts_tablet_manager.cc:1397] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.107s user 0.007s sys 0.011s
I20250115 14:52:04.116948 1168 raft_consensus.cc:357] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.117187 1168 raft_consensus.cc:383] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.117281 1168 raft_consensus.cc:738] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.117545 1168 consensus_queue.cc:260] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.118499 1164 tablet_bootstrap.cc:492] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:04.118652 1164 ts_tablet_manager.cc:1397] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.131s user 0.012s sys 0.003s
I20250115 14:52:04.119336 1164 raft_consensus.cc:357] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:04.120040 847 consensus_peers.cc:487] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3fbde011216426cb41c51fe359dcb5f. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.119509 1164 raft_consensus.cc:383] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.120790 1164 raft_consensus.cc:738] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.121088 1168 ts_tablet_manager.cc:1428] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250115 14:52:04.121059 1164 consensus_queue.cc:260] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.121544 1164 ts_tablet_manager.cc:1428] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:04.121326 1751 ts_tablet_manager.cc:1939] T 0e924821d12045f6bc658461675aead4 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:04.122877 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:04.124434 1164 tablet_bootstrap.cc:492] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:04.125770 1164 tablet_bootstrap.cc:654] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.126443 879 tablet_service.cc:1514] Processing DeleteTablet for tablet d195be2d543f4a8693045a024912c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.127199 1025 tablet_service.cc:1467] Processing CreateTablet for tablet e9febafb7e354e289cd3f864311f14cb (DEFAULT_TABLE table=default.Table_c72e28daf2c44e519f2e934f811954e1 [id=9fd8139f0247410d8aad7dcb63901905]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.127594 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9febafb7e354e289cd3f864311f14cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.129842 450 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 63f57c1af0b541d992e6f0f9b02fc554
I20250115 14:52:04.130192 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e180e37a1d5d46d78380323662b80f85
I20250115 14:52:04.130581 451 catalog_manager.cc:6596] ReplaceTablet: tablet e180e37a1d5d46d78380323662b80f85 deleted and replaced by tablet 1a676807ef38485ea49b0e92f5dc9862
I20250115 14:52:04.130981 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet e180e37a1d5d46d78380323662b80f85 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a676807ef38485ea49b0e92f5dc9862) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.133940 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 40eae66b35974524992b2ef943273291 (DEFAULT_TABLE table=default.Table_c16ddd6216634793b8438ac0421e8cb5 [id=cc922c9c1be140ce91ef4f8d5170ac66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.134117 879 tablet_service.cc:1514] Processing DeleteTablet for tablet e180e37a1d5d46d78380323662b80f85 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a676807ef38485ea49b0e92f5dc9862) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.134299 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40eae66b35974524992b2ef943273291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.135572 879 tablet_service.cc:1467] Processing CreateTablet for tablet 40eae66b35974524992b2ef943273291 (DEFAULT_TABLE table=default.Table_c16ddd6216634793b8438ac0421e8cb5 [id=cc922c9c1be140ce91ef4f8d5170ac66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.135887 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40eae66b35974524992b2ef943273291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.136776 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d8fd4b26294e8ca6745b95f2e03231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.140503 1876 raft_consensus.cc:491] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.140625 1876 raft_consensus.cc:513] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.141204 1876 leader_election.cc:290] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.141644 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84bc24a71e431cb47b93f2b0dcaa90" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:04.142809 983 leader_election.cc:343] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.143067 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84bc24a71e431cb47b93f2b0dcaa90" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:04.143553 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2918791b551d4aecb80aee6a40351a45" } modify_external_catalogs: true
I20250115 14:52:04.143752 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2918791b551d4aecb80aee6a40351a45" } modify_external_catalogs: true
I20250115 14:52:04.144256 1168 tablet_bootstrap.cc:492] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:04.145773 1168 tablet_bootstrap.cc:654] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:04.146915 984 leader_election.cc:343] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.147024 984 leader_election.cc:304] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.147423 1876 raft_consensus.cc:2743] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.148662 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c1a77d07b1c946b18e0769a37d8a50a1
I20250115 14:52:04.150326 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a77d07b1c946b18e0769a37d8a50a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.150960 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]). Expected version 1 got 0
I20250115 14:52:04.151683 879 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a77d07b1c946b18e0769a37d8a50a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.154134 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3f8db86f4b5b4f1b88804f296cc86c3f" } modify_external_catalogs: true
I20250115 14:52:04.154366 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3f8db86f4b5b4f1b88804f296cc86c3f" } modify_external_catalogs: true
I20250115 14:52:04.154552 2110 raft_consensus.cc:2798] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:52:04.154611 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 124172us
I20250115 14:52:04.154680 2110 raft_consensus.cc:695] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:04.154904 2110 consensus_queue.cc:237] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.155151 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]) successfully deleted
I20250115 14:52:04.155803 1019 tablet_service.cc:1467] Processing CreateTablet for tablet edb5b4911c70478f8d02e476e9cb1769 (DEFAULT_TABLE table=default.Table_f418504cc929429c936bfb16aa538464 [id=36109f49d7b54a2aab30697c22af6386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.155958 1955 raft_consensus.cc:491] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.156069 1955 raft_consensus.cc:513] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.156487 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb5b4911c70478f8d02e476e9cb1769. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.157295 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff71ded3b074ff0979e2090131f4eaa (DEFAULT_TABLE table=default.Table_dc4acc7152c24a22a0e7ac7de1943df3 [id=d5c34a4a002c42dd8bd29c970833542d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.157627 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff71ded3b074ff0979e2090131f4eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.159765 883 tablet_service.cc:1467] Processing CreateTablet for tablet c1a77d07b1c946b18e0769a37d8a50a1 (DEFAULT_TABLE table=default.Table_2918791b551d4aecb80aee6a40351a45 [id=ddff2ff64a44471680d78cbd58afbf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.160085 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1a77d07b1c946b18e0769a37d8a50a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.160516 879 tablet_service.cc:1467] Processing CreateTablet for tablet edb5b4911c70478f8d02e476e9cb1769 (DEFAULT_TABLE table=default.Table_f418504cc929429c936bfb16aa538464 [id=36109f49d7b54a2aab30697c22af6386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.160827 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb5b4911c70478f8d02e476e9cb1769. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.161343 1955 leader_election.cc:290] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.166280 889 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff71ded3b074ff0979e2090131f4eaa (DEFAULT_TABLE table=default.Table_dc4acc7152c24a22a0e7ac7de1943df3 [id=d5c34a4a002c42dd8bd29c970833542d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.166401 1025 tablet_service.cc:1467] Processing CreateTablet for tablet c1a77d07b1c946b18e0769a37d8a50a1 (DEFAULT_TABLE table=default.Table_2918791b551d4aecb80aee6a40351a45 [id=ddff2ff64a44471680d78cbd58afbf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.166720 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1a77d07b1c946b18e0769a37d8a50a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.167009 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff71ded3b074ff0979e2090131f4eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.170724 883 tablet_service.cc:1467] Processing CreateTablet for tablet 6f924700fa524bae96016b039ab3b819 (DEFAULT_TABLE table=default.Table_e541499a248445d7929d628f87f2d65e [id=846f3223c6aa4b38bb567dd5e05e03c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.171038 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f924700fa524bae96016b039ab3b819. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.174925 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 11fddbd9ba14443a91865387e1d1c238
I20250115 14:52:04.175606 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fddbd9ba14443a91865387e1d1c238 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.176554 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fddbd9ba14443a91865387e1d1c238 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.177580 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]). Expected version 1 got 0
I20250115 14:52:04.178884 1167 ts_tablet_manager.cc:1918] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.178989 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 6f924700fa524bae96016b039ab3b819 (DEFAULT_TABLE table=default.Table_e541499a248445d7929d628f87f2d65e [id=846f3223c6aa4b38bb567dd5e05e03c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.179338 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f924700fa524bae96016b039ab3b819. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.179693 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 4e459144038d411d9e76322dee8f004b (DEFAULT_TABLE table=default.Table_349a30fe26984a52913598e0f22c9295 [id=6fb7f3061f0041e0a834ad48fdc1f8e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.180023 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e459144038d411d9e76322dee8f004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.183030 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d8fd4b26294e8ca6745b95f2e03231" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:04.184212 741 tablet_service.cc:1514] Processing DeleteTablet for tablet e198b4e1a65b4704bf2bb8f7d5987385 with delete_type TABLET_DATA_DELETED (replaced by tablet 65a6f83730f74d13823bd15d93538df7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.184389 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 4821224ac8f1472d8b1e8afc06a118a3 with delete_type TABLET_DATA_DELETED (replaced by tablet 63f57c1af0b541d992e6f0f9b02fc554) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.184553 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4c2707915e4e97ad6c66cb249c6d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6afbdd0b585e4e7e82067dce061924d0) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.184909 745 tablet_service.cc:1467] Processing CreateTablet for tablet 8d4c30bccb3549e6963dfbbecf68cbbe (DEFAULT_TABLE table=default.Table_9e128687957b4b559bb77fd82039e563 [id=856ea3ceeee84f3283d50a003e582724]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.185225 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d4c30bccb3549e6963dfbbecf68cbbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.186926 1164 tablet_bootstrap.cc:492] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:04.187047 1164 ts_tablet_manager.cc:1397] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250115 14:52:04.187685 1164 raft_consensus.cc:357] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.187844 1164 raft_consensus.cc:383] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.187891 1164 raft_consensus.cc:738] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.188104 1164 consensus_queue.cc:260] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.188364 739 tablet_service.cc:1467] Processing CreateTablet for tablet e9febafb7e354e289cd3f864311f14cb (DEFAULT_TABLE table=default.Table_c72e28daf2c44e519f2e934f811954e1 [id=9fd8139f0247410d8aad7dcb63901905]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.188585 1164 ts_tablet_manager.cc:1428] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250115 14:52:04.188645 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9febafb7e354e289cd3f864311f14cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.188832 1164 tablet_bootstrap.cc:492] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:04.190054 1164 tablet_bootstrap.cc:654] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.191558 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7e33fc39d94c6286060f1e02f02372 (DEFAULT_TABLE table=default.Table_e32df962655644d4b4a7e3ba02623ed3 [id=365be1cda6834cfc9730006004374444]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.191876 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d7e33fc39d94c6286060f1e02f02372. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.192533 743 tablet_service.cc:1467] Processing CreateTablet for tablet 40eae66b35974524992b2ef943273291 (DEFAULT_TABLE table=default.Table_c16ddd6216634793b8438ac0421e8cb5 [id=cc922c9c1be140ce91ef4f8d5170ac66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.192832 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40eae66b35974524992b2ef943273291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.196555 750 tablet_service.cc:1467] Processing CreateTablet for tablet edb5b4911c70478f8d02e476e9cb1769 (DEFAULT_TABLE table=default.Table_f418504cc929429c936bfb16aa538464 [id=36109f49d7b54a2aab30697c22af6386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.196839 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb5b4911c70478f8d02e476e9cb1769. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.197341 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 0e924821d12045f6bc658461675aead4 (table default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]) successfully deleted
I20250115 14:52:04.198450 1751 tablet_replica.cc:331] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:04.198746 1751 raft_consensus.cc:2238] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.199113 1751 raft_consensus.cc:2267] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.199275 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 29adc03e38624dfc910e07de24466d1b (DEFAULT_TABLE table=default.Table_9c613e4019394b9790eb4d917dc249c5 [id=023cc9b8568241768e71efe6197f7cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.199564 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29adc03e38624dfc910e07de24466d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.199936 1751 ts_tablet_manager.cc:1905] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.200515 749 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff71ded3b074ff0979e2090131f4eaa (DEFAULT_TABLE table=default.Table_dc4acc7152c24a22a0e7ac7de1943df3 [id=d5c34a4a002c42dd8bd29c970833542d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.200799 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff71ded3b074ff0979e2090131f4eaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.204627 751 tablet_service.cc:1514] Processing DeleteTablet for tablet c1a77d07b1c946b18e0769a37d8a50a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.204821 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e180e37a1d5d46d78380323662b80f85 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a676807ef38485ea49b0e92f5dc9862) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.204990 755 tablet_service.cc:1467] Processing CreateTablet for tablet c1a77d07b1c946b18e0769a37d8a50a1 (DEFAULT_TABLE table=default.Table_2918791b551d4aecb80aee6a40351a45 [id=ddff2ff64a44471680d78cbd58afbf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.205260 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1a77d07b1c946b18e0769a37d8a50a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.205094 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_57cdcb46b45d430b84affd591c061ced"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.205745 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57cdcb46b45d430b84affd591c061ced in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.215888 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_56da29ad1fe24546bc5c6a4a17a774e7" } modify_external_catalogs: true
I20250115 14:52:04.216070 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_56da29ad1fe24546bc5c6a4a17a774e7" } modify_external_catalogs: true
I20250115 14:52:04.217273 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5f4c7f8a9cd745ea935c1997f991a1f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.217775 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f4c7f8a9cd745ea935c1997f991a1f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.219065 739 tablet_service.cc:1467] Processing CreateTablet for tablet 6f924700fa524bae96016b039ab3b819 (DEFAULT_TABLE table=default.Table_e541499a248445d7929d628f87f2d65e [id=846f3223c6aa4b38bb567dd5e05e03c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.219408 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f924700fa524bae96016b039ab3b819. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.224426 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ed777d7496514ec5986a7e6f53d532b1
I20250115 14:52:04.224539 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4e459144038d411d9e76322dee8f004b (DEFAULT_TABLE table=default.Table_349a30fe26984a52913598e0f22c9295 [id=6fb7f3061f0041e0a834ad48fdc1f8e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.224860 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e459144038d411d9e76322dee8f004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.225672 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ec36b5656e984a2a9468f7d11f75d663"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.226172 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec36b5656e984a2a9468f7d11f75d663 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.227649 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fddbd9ba14443a91865387e1d1c238 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.230542 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dd6e49ce9dc4402b88b42b9ed72e08eb" } new_table_name: "default.Table_65167815994b4ed4aa3d30cb8b168d63" modify_external_catalogs: true
I20250115 14:52:04.235371 2041 tablet.cc:1721] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:04.238521 739 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7e33fc39d94c6286060f1e02f02372 (DEFAULT_TABLE table=default.Table_e32df962655644d4b4a7e3ba02623ed3 [id=365be1cda6834cfc9730006004374444]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.238860 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d7e33fc39d94c6286060f1e02f02372. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.239310 755 tablet_service.cc:1514] Processing DeleteTablet for tablet ed777d7496514ec5986a7e6f53d532b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.240007 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13b4c0edc2c4fc9a1c527e1ff6a3d00" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:04.240216 1047 raft_consensus.cc:2388] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d7ecece2ab49e1be7079216517d13b in current term 1: Already voted for candidate 73c86158cc2641f182ea50a06634592d in this term.
I20250115 14:52:04.242766 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7]). Expected version 2 got 0
I20250115 14:52:04.243033 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_7b4753acbcca4cdc94831798b3d47baa [id=7628dbb6818f443680a97fef665d6d2b]). Expected version 2 got 0
W20250115 14:52:04.243391 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6cb8dfa7c42480bbd139d5acf059a57 consensus state has no leader
I20250115 14:52:04.244284 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_306d0c66a91b4c7784d68f7649a7dd69" } new_table_name: "default.Table_90c834d5ef634e6aa797dd220e3d5faa" modify_external_catalogs: true
I20250115 14:52:04.248415 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 83cd48dfd1084d498153dfa11c0ab858 (DEFAULT_TABLE table=default.Table_edd5bd0ae3dd4bd994b4e746c1c14c9a [id=8281493387d2428fb5a95bfd586d48e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.248770 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cd48dfd1084d498153dfa11c0ab858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.251668 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_32080fc25c67476db569c3357f008e2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.252172 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32080fc25c67476db569c3357f008e2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.254767 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_803f17ac9c4e40ffb06dc6ac0faaf7ed" } new_table_name: "default.Table_6ca072719bcc42e980d83a7586939221" modify_external_catalogs: true
I20250115 14:52:04.259032 755 tablet_service.cc:1467] Processing CreateTablet for tablet 105b6148ae944e5c906e07d987177cc5 (DEFAULT_TABLE table=default.Table_afd9df7559aa485789c8e7bd5d43c3ef [id=15a1f705007143a4a8230e87e3042265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.259166 743 tablet_service.cc:1467] Processing CreateTablet for tablet 338706b2d7514f94afd3579da458b4ec (DEFAULT_TABLE table=default.Table_41d5e114b3da4fca97f9efc5a69dfe2f [id=ce80579d2d2f4d8e86971514f6e53454]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.259694 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 338706b2d7514f94afd3579da458b4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.259832 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 105b6148ae944e5c906e07d987177cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.260957 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ed777d7496514ec5986a7e6f53d532b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.264860 750 tablet_service.cc:1467] Processing CreateTablet for tablet 83cd48dfd1084d498153dfa11c0ab858 (DEFAULT_TABLE table=default.Table_edd5bd0ae3dd4bd994b4e746c1c14c9a [id=8281493387d2428fb5a95bfd586d48e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.265237 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cd48dfd1084d498153dfa11c0ab858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.265726 2110 raft_consensus.cc:491] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.265888 2110 raft_consensus.cc:513] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.265821 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 338706b2d7514f94afd3579da458b4ec (DEFAULT_TABLE table=default.Table_41d5e114b3da4fca97f9efc5a69dfe2f [id=ce80579d2d2f4d8e86971514f6e53454]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.266469 2110 leader_election.cc:290] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.267040 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 338706b2d7514f94afd3579da458b4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.267367 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb19d39258e444799d403a43475d9c8a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:04.267560 1047 raft_consensus.cc:2463] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:04.268476 710 leader_election.cc:304] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:04.268791 2110 raft_consensus.cc:2798] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:04.268734 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 105b6148ae944e5c906e07d987177cc5 (DEFAULT_TABLE table=default.Table_afd9df7559aa485789c8e7bd5d43c3ef [id=15a1f705007143a4a8230e87e3042265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.268924 2110 raft_consensus.cc:491] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:04.269047 2110 raft_consensus.cc:3054] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.269064 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 105b6148ae944e5c906e07d987177cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.269723 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3ae69bc8a5ee4225a1952f93b535c49b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.269790 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5c0d0b9700e4413d85d5a1dbc9a23a45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.270421 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ae69bc8a5ee4225a1952f93b535c49b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.270702 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ec9e96994cca43569c5976b968296550"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.271112 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c0d0b9700e4413d85d5a1dbc9a23a45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:04.271303 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec9e96994cca43569c5976b968296550 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.273422 1166 ts_tablet_manager.cc:1918] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.273521 1166 log.cc:1198] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/caed629878594954842dfc6a40cc8b6d
I20250115 14:52:04.273480 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7b4753acbcca4cdc94831798b3d47baa" } new_table_name: "default.Table_a08bf8546b9148dc97c582984b4409c8" modify_external_catalogs: true
I20250115 14:52:04.273540 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_12287ed111bb450dbe84bdb433fc746e" } new_table_name: "default.Table_0ee70516fe3c4a12983e1448718e7943" modify_external_catalogs: true
I20250115 14:52:04.273785 1166 ts_tablet_manager.cc:1939] T caed629878594954842dfc6a40cc8b6d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:04.274972 1166 tablet_replica.cc:331] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:04.275147 1166 raft_consensus.cc:2238] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.275276 1166 raft_consensus.cc:2267] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.275738 1166 ts_tablet_manager.cc:1905] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.279129 2110 raft_consensus.cc:513] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.279711 2110 leader_election.cc:290] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.282644 755 tablet_service.cc:1467] Processing CreateTablet for tablet f1951b7ea6c0492b9920836515e7e5e5 (DEFAULT_TABLE table=default.Table_5ef75e2dec224af78b8b5da8cf17abfe [id=77b40c1e0b8047f5a785ecf7dc2fdc53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.283838 1015 tablet_service.cc:1467] Processing CreateTablet for tablet f1951b7ea6c0492b9920836515e7e5e5 (DEFAULT_TABLE table=default.Table_5ef75e2dec224af78b8b5da8cf17abfe [id=77b40c1e0b8047f5a785ecf7dc2fdc53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.284369 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1951b7ea6c0492b9920836515e7e5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.287683 1019 tablet_service.cc:1467] Processing CreateTablet for tablet e7315a41489a4f8296ad8b12c039b0a2 (DEFAULT_TABLE table=default.Table_f7ef4548b4c2429ea9bfa1fea3d621fe [id=795c8172f7b946babddf8ee7c2707582]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.288007 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7315a41489a4f8296ad8b12c039b0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.288971 1162 tablet_bootstrap.cc:492] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:04.289088 1162 ts_tablet_manager.cc:1397] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.184s user 0.004s sys 0.033s
I20250115 14:52:04.290004 1162 raft_consensus.cc:357] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.290100 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet caed629878594954842dfc6a40cc8b6d (table default.Table_544d606cf9af4accb670d91d7d717948 [id=b6aeb933c5e64e88a6d794a98650acbe]) successfully deleted
I20250115 14:52:04.290179 1162 raft_consensus.cc:383] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.290222 1162 raft_consensus.cc:738] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
W20250115 14:52:04.290323 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 02ce16fea4f9469c9fc06fa24f6554be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02ce16fea4f9469c9fc06fa24f6554be already in progress: creating tablet
I20250115 14:52:04.290421 1162 consensus_queue.cc:260] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:04.290517 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet be63bae96c834a8bbbb1174c027935a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be63bae96c834a8bbbb1174c027935a2 already in progress: creating tablet
I20250115 14:52:04.288759 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb19d39258e444799d403a43475d9c8a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:04.292912 1047 raft_consensus.cc:3054] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.293871 1162 ts_tablet_manager.cc:1428] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:04.293864 741 tablet_service.cc:1467] Processing CreateTablet for tablet 3e4b8b77c11a4582b0d6b85f4c49aa82 (DEFAULT_TABLE table=default.Table_24d51c1e7d8d4fdd83f72d6094af1d6b [id=8e3ab65e00e54c92a2e1bb98163c77ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.294124 1162 tablet_bootstrap.cc:492] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:04.295229 1047 raft_consensus.cc:2463] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:04.296798 745 tablet_service.cc:1467] Processing CreateTablet for tablet e7315a41489a4f8296ad8b12c039b0a2 (DEFAULT_TABLE table=default.Table_f7ef4548b4c2429ea9bfa1fea3d621fe [id=795c8172f7b946babddf8ee7c2707582]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.297833 452 master_service.cc:946] ReplaceTablet: received request to replace tablet a915e0c0a30940ed8a6233f117e2a560 from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:04.298142 984 consensus_peers.cc:487] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e3ec000089a47a2930dec382192cd6f. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.300748 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 13053264b5ca41dd85e53d928aa511fd (DEFAULT_TABLE table=default.Table_0195de43d08f499d99237ab07b24b892 [id=c3a1109adf97490b965beae0f8d78322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.301064 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13053264b5ca41dd85e53d928aa511fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.300947 749 tablet_service.cc:1467] Processing CreateTablet for tablet 29adc03e38624dfc910e07de24466d1b (DEFAULT_TABLE table=default.Table_9c613e4019394b9790eb4d917dc249c5 [id=023cc9b8568241768e71efe6197f7cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.301752 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 496dec75500347699625a755dceed2bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.307752 1751 ts_tablet_manager.cc:1918] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:04.307853 1751 log.cc:1198] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e9e5bc6131944fcab9c550caac4f8d85
I20250115 14:52:04.308158 1751 ts_tablet_manager.cc:1939] T e9e5bc6131944fcab9c550caac4f8d85 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:04.308419 2110 raft_consensus.cc:491] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.308645 2110 raft_consensus.cc:513] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.309165 1162 tablet_bootstrap.cc:654] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.309661 2110 leader_election.cc:290] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.309931 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4b8b77c11a4582b0d6b85f4c49aa82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.310288 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1951b7ea6c0492b9920836515e7e5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.310494 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7315a41489a4f8296ad8b12c039b0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.314555 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 919ab6ec6f3745e082aa441dfebc235e (DEFAULT_TABLE table=default.Table_9b06f855e23e47cbb239075f101b585c [id=d6f2abb6e8894bf58f3aa4876cc38ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.315119 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 919ab6ec6f3745e082aa441dfebc235e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.322767 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4238c63c4c42aba30f906a87c644a2 (DEFAULT_TABLE table=default.Table_0b2109f1c1814a63a365822b2102e61e [id=96b3610ffe9647e2ae452779d96fef96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.323315 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a4238c63c4c42aba30f906a87c644a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.327929 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29adc03e38624dfc910e07de24466d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.180402 882 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7e33fc39d94c6286060f1e02f02372 (DEFAULT_TABLE table=default.Table_e32df962655644d4b4a7e3ba02623ed3 [id=365be1cda6834cfc9730006004374444]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.336020 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d7e33fc39d94c6286060f1e02f02372. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.324265 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 646) took 1122 ms. Trace:
I20250115 14:52:04.336210 1166 ts_tablet_manager.cc:1918] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.336170 1751 rpcz_store.cc:276] 0115 14:52:03.201480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.201718 (+ 238us) service_pool.cc:225] Handling call
0115 14:52:04.198355 (+996637us) ts_tablet_manager.cc:1145] Deleting tablet e9e5bc6131944fcab9c550caac4f8d85
0115 14:52:04.198390 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.258186 (+ 59796us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.307652 (+ 49466us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.324241 (+ 16589us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":996518}
I20250115 14:52:04.336802 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e9e5bc6131944fcab9c550caac4f8d85 (table default.Table_609197916c9b4c81aebdaf70e78cc436 [id=33e3df28f2d445c5ba4e8b0ce4458c1a]) successfully deleted
I20250115 14:52:04.337023 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 647) took 1096 ms. Trace:
I20250115 14:52:04.337136 1751 rpcz_store.cc:276] 0115 14:52:03.240559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.240623 (+ 64us) service_pool.cc:225] Handling call
0115 14:52:04.336831 (+1096208us) ts_tablet_manager.cc:1145] Deleting tablet f3280e8ba6db4446aa3e3bfe2836abfa
0115 14:52:04.336841 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.337007 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1096002}
W20250115 14:52:04.337541 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f3280e8ba6db4446aa3e3bfe2836abfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3280e8ba6db4446aa3e3bfe2836abfa already in progress: creating tablet
I20250115 14:52:04.337622 1751 tablet_replica.cc:331] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:04.337914 1751 raft_consensus.cc:2238] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:04.338371 1751 raft_consensus.cc:2267] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.339004 1025 tablet_service.cc:1467] Processing CreateTablet for tablet fb81e8fcc38a41e69cfb0e34dbaaac84 (DEFAULT_TABLE table=default.Table_ae36f7ba541a43ffad08ed6fc6b10ec0 [id=349052fac9f94319b3b24fce63bb3a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.339210 1751 ts_tablet_manager.cc:1905] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.339329 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb81e8fcc38a41e69cfb0e34dbaaac84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.343361 1166 log.cc:1198] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b2c385edf4414157845b971aaf366831
I20250115 14:52:04.343681 1166 ts_tablet_manager.cc:1939] T b2c385edf4414157845b971aaf366831 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:04.344723 1166 tablet_replica.cc:331] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:04.181272 883 tablet_service.cc:1467] Processing CreateTablet for tablet 4e459144038d411d9e76322dee8f004b (DEFAULT_TABLE table=default.Table_349a30fe26984a52913598e0f22c9295 [id=6fb7f3061f0041e0a834ad48fdc1f8e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.344921 1166 raft_consensus.cc:2238] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.345055 1166 raft_consensus.cc:2267] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.345093 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e459144038d411d9e76322dee8f004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.345533 1166 ts_tablet_manager.cc:1905] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.198308 878 tablet_service.cc:1467] Processing CreateTablet for tablet 29adc03e38624dfc910e07de24466d1b (DEFAULT_TABLE table=default.Table_9c613e4019394b9790eb4d917dc249c5 [id=023cc9b8568241768e71efe6197f7cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.346863 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29adc03e38624dfc910e07de24466d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.347469 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 2edd5570ca604ac6bd85686e0e708187 (DEFAULT_TABLE table=default.Table_5c73eff7b16a4a11ad47cd4c32a6fd0a [id=c1dc41883d4448d882e8acc833ebc260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.347787 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2edd5570ca604ac6bd85686e0e708187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.347800 710 leader_election.cc:304] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:04.348135 2110 raft_consensus.cc:2798] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:04.348235 2110 raft_consensus.cc:695] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:04.198952 887 tablet_service.cc:1467] Processing CreateTablet for tablet 338706b2d7514f94afd3579da458b4ec (DEFAULT_TABLE table=default.Table_41d5e114b3da4fca97f9efc5a69dfe2f [id=ce80579d2d2f4d8e86971514f6e53454]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.348531 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 338706b2d7514f94afd3579da458b4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.348460 2110 consensus_queue.cc:237] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.350667 1162 tablet_bootstrap.cc:492] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:04.350790 1162 ts_tablet_manager.cc:1397] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.001s
I20250115 14:52:04.351471 1162 raft_consensus.cc:357] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.351656 1162 raft_consensus.cc:383] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.351708 1162 raft_consensus.cc:738] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.229447 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ed777d7496514ec5986a7e6f53d532b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.351891 1162 consensus_queue.cc:260] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.352407 1162 ts_tablet_manager.cc:1428] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:04.234443 2079 tablet.cc:1721] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:04.246456 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13b4c0edc2c4fc9a1c527e1ff6a3d00" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:04.246671 848 leader_election.cc:343] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.353515 848 leader_election.cc:304] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.353534 741 tablet_service.cc:1467] Processing CreateTablet for tablet fb81e8fcc38a41e69cfb0e34dbaaac84 (DEFAULT_TABLE table=default.Table_ae36f7ba541a43ffad08ed6fc6b10ec0 [id=349052fac9f94319b3b24fce63bb3a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.353744 2013 raft_consensus.cc:2743] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:04.353813 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 107145us
W20250115 14:52:04.354048 848 consensus_peers.cc:487] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d97c2f443ab040f9bc1924f45379165e. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.354110 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb81e8fcc38a41e69cfb0e34dbaaac84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.264816 1168 tablet_bootstrap.cc:492] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:04.355142 1168 ts_tablet_manager.cc:1397] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.211s user 0.014s sys 0.009s
I20250115 14:52:04.355499 879 tablet_service.cc:1467] Processing CreateTablet for tablet 83cd48dfd1084d498153dfa11c0ab858 (DEFAULT_TABLE table=default.Table_edd5bd0ae3dd4bd994b4e746c1c14c9a [id=8281493387d2428fb5a95bfd586d48e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.355922 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83cd48dfd1084d498153dfa11c0ab858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.355871 1168 raft_consensus.cc:357] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.356029 1168 raft_consensus.cc:383] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.356072 1168 raft_consensus.cc:738] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.356222 1168 consensus_queue.cc:260] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.356523 884 tablet_service.cc:1467] Processing CreateTablet for tablet f1951b7ea6c0492b9920836515e7e5e5 (DEFAULT_TABLE table=default.Table_5ef75e2dec224af78b8b5da8cf17abfe [id=77b40c1e0b8047f5a785ecf7dc2fdc53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.356689 1168 ts_tablet_manager.cc:1428] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250115 14:52:04.356808 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1951b7ea6c0492b9920836515e7e5e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.356920 1168 tablet_bootstrap.cc:492] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:04.359156 874 tablet_service.cc:1467] Processing CreateTablet for tablet e7315a41489a4f8296ad8b12c039b0a2 (DEFAULT_TABLE table=default.Table_f7ef4548b4c2429ea9bfa1fea3d621fe [id=795c8172f7b946babddf8ee7c2707582]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.359433 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7315a41489a4f8296ad8b12c039b0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.359681 739 tablet_service.cc:1467] Processing CreateTablet for tablet 13053264b5ca41dd85e53d928aa511fd (DEFAULT_TABLE table=default.Table_0195de43d08f499d99237ab07b24b892 [id=c3a1109adf97490b965beae0f8d78322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.359970 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13053264b5ca41dd85e53d928aa511fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.360522 880 tablet_service.cc:1467] Processing CreateTablet for tablet 13053264b5ca41dd85e53d928aa511fd (DEFAULT_TABLE table=default.Table_0195de43d08f499d99237ab07b24b892 [id=c3a1109adf97490b965beae0f8d78322]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.360816 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13053264b5ca41dd85e53d928aa511fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.362823 1876 raft_consensus.cc:491] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.362938 1876 raft_consensus.cc:513] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.363070 879 tablet_service.cc:1467] Processing CreateTablet for tablet 0f82d944a7e04ace8e4addb3897d2c8e (DEFAULT_TABLE table=default.Table_d03b53f6993c4f3aae49c2155890bfc0 [id=42f0e3311f664194b8347878ac195ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.363418 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f82d944a7e04ace8e4addb3897d2c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.363526 1876 leader_election.cc:290] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.363853 875 tablet_service.cc:1467] Processing CreateTablet for tablet 3e4b8b77c11a4582b0d6b85f4c49aa82 (DEFAULT_TABLE table=default.Table_24d51c1e7d8d4fdd83f72d6094af1d6b [id=8e3ab65e00e54c92a2e1bb98163c77ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.364180 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4b8b77c11a4582b0d6b85f4c49aa82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.364519 886 tablet_service.cc:1467] Processing CreateTablet for tablet 919ab6ec6f3745e082aa441dfebc235e (DEFAULT_TABLE table=default.Table_9b06f855e23e47cbb239075f101b585c [id=d6f2abb6e8894bf58f3aa4876cc38ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.364615 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b2c385edf4414157845b971aaf366831 (table default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]) successfully deleted
I20250115 14:52:04.364799 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 919ab6ec6f3745e082aa441dfebc235e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:04.365067 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb already in progress: creating tablet
I20250115 14:52:04.365844 1162 tablet_bootstrap.cc:492] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:04.367643 876 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4238c63c4c42aba30f906a87c644a2 (DEFAULT_TABLE table=default.Table_0b2109f1c1814a63a365822b2102e61e [id=96b3610ffe9647e2ae452779d96fef96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.367933 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a4238c63c4c42aba30f906a87c644a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.368006 1162 tablet_bootstrap.cc:654] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.370260 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 0f82d944a7e04ace8e4addb3897d2c8e (DEFAULT_TABLE table=default.Table_d03b53f6993c4f3aae49c2155890bfc0 [id=42f0e3311f664194b8347878ac195ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.371163 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9853dcdfee5f43de84798cfa0bef674e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:04.371685 879 tablet_service.cc:1467] Processing CreateTablet for tablet 6b28a180aa0e4ef2b62890233cd512a4 (DEFAULT_TABLE table=default.Table_00e45525259140518db638102c114d02 [id=aa031221a7034b62a85c8914d6872ba7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:04.371872 984 leader_election.cc:343] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.371958 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b28a180aa0e4ef2b62890233cd512a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.372519 885 tablet_service.cc:1467] Processing CreateTablet for tablet fb81e8fcc38a41e69cfb0e34dbaaac84 (DEFAULT_TABLE table=default.Table_ae36f7ba541a43ffad08ed6fc6b10ec0 [id=349052fac9f94319b3b24fce63bb3a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.372807 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb81e8fcc38a41e69cfb0e34dbaaac84. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:04.373046 710 leader_election.cc:343] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 496dec75500347699625a755dceed2bb
W20250115 14:52:04.374257 848 consensus_peers.cc:487] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3fbde011216426cb41c51fe359dcb5f. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.374823 1168 tablet_bootstrap.cc:654] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.374809 755 tablet_service.cc:1467] Processing CreateTablet for tablet 2edd5570ca604ac6bd85686e0e708187 (DEFAULT_TABLE table=default.Table_5c73eff7b16a4a11ad47cd4c32a6fd0a [id=c1dc41883d4448d882e8acc833ebc260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.375118 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2edd5570ca604ac6bd85686e0e708187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.268373 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb19d39258e444799d403a43475d9c8a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:04.289253 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb19d39258e444799d403a43475d9c8a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:04.377003 911 raft_consensus.cc:2463] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
W20250115 14:52:04.304569 847 consensus_peers.cc:487] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3a7a487a42949bd9641c38d6c1fcfd3. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:04.377182 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 72650.6us
I20250115 14:52:04.377816 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496dec75500347699625a755dceed2bb" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:04.377977 908 raft_consensus.cc:2463] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:04.378664 710 leader_election.cc:304] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:04.378922 2117 raft_consensus.cc:3054] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.381215 2117 raft_consensus.cc:2743] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.381963 710 leader_election.cc:304] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:04.382300 1876 raft_consensus.cc:491] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.382408 1876 raft_consensus.cc:513] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.382965 1876 leader_election.cc:290] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.383524 2117 raft_consensus.cc:2798] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:04.383595 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e842e1c2b44754b3b9a2e026486a84" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:04.383949 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e842e1c2b44754b3b9a2e026486a84" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:04.384053 741 tablet_service.cc:1467] Processing CreateTablet for tablet 0f82d944a7e04ace8e4addb3897d2c8e (DEFAULT_TABLE table=default.Table_d03b53f6993c4f3aae49c2155890bfc0 [id=42f0e3311f664194b8347878ac195ed9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:04.384375 984 leader_election.cc:343] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:04.384568 983 leader_election.cc:343] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.384645 983 leader_election.cc:304] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.384694 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f82d944a7e04ace8e4addb3897d2c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.385041 1876 raft_consensus.cc:2743] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.383595 2117 raft_consensus.cc:491] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:04.386068 2117 raft_consensus.cc:3054] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.389678 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f82d944a7e04ace8e4addb3897d2c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.391754 1166 ts_tablet_manager.cc:1918] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.391852 1166 log.cc:1198] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/bc8337d181d44ac7bb82551a677b6831
I20250115 14:52:04.392115 1166 ts_tablet_manager.cc:1939] T bc8337d181d44ac7bb82551a677b6831 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:04.394166 1162 tablet_bootstrap.cc:492] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:04.394289 1162 ts_tablet_manager.cc:1397] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250115 14:52:04.394923 1162 raft_consensus.cc:357] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.395087 1162 raft_consensus.cc:383] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.395139 1162 raft_consensus.cc:738] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.395431 1162 consensus_queue.cc:260] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.396637 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet bc8337d181d44ac7bb82551a677b6831 (table default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]) successfully deleted
I20250115 14:52:04.397305 1162 ts_tablet_manager.cc:1428] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:04.397372 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6b28a180aa0e4ef2b62890233cd512a4 (DEFAULT_TABLE table=default.Table_00e45525259140518db638102c114d02 [id=aa031221a7034b62a85c8914d6872ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.397742 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b28a180aa0e4ef2b62890233cd512a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.397936 1162 tablet_bootstrap.cc:492] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:04.398911 1166 tablet_replica.cc:331] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:04.399091 1166 raft_consensus.cc:2238] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:04.399319 1166 pending_rounds.cc:62] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:04.399367 1166 pending_rounds.cc:69] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558358141116416 op_type: NO_OP noop_request { }
I20250115 14:52:04.399467 1166 raft_consensus.cc:2883] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:04.399544 1166 raft_consensus.cc:2267] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.400837 891 tablet_service.cc:1467] Processing CreateTablet for tablet 105b6148ae944e5c906e07d987177cc5 (DEFAULT_TABLE table=default.Table_afd9df7559aa485789c8e7bd5d43c3ef [id=15a1f705007143a4a8230e87e3042265]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.401167 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 105b6148ae944e5c906e07d987177cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.403609 2117 raft_consensus.cc:513] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.390879 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9853dcdfee5f43de84798cfa0bef674e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:04.404675 983 leader_election.cc:343] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.404815 983 leader_election.cc:304] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.405184 1876 raft_consensus.cc:2743] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.406004 1167 log.cc:1198] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/276abefcab6a422aa6467f00864c01f4
I20250115 14:52:04.406355 1167 ts_tablet_manager.cc:1939] T 276abefcab6a422aa6467f00864c01f4 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:04.406693 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 567) took 2436 ms. Trace:
I20250115 14:52:04.406817 1167 rpcz_store.cc:276] 0115 14:52:01.970147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:01.970199 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:04.107962 (+2137763us) ts_tablet_manager.cc:1145] Deleting tablet 276abefcab6a422aa6467f00864c01f4
0115 14:52:04.107971 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.146340 (+ 38369us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.178808 (+ 32468us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.406680 (+227872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2137618}
I20250115 14:52:04.406886 1162 tablet_bootstrap.cc:654] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.407321 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 276abefcab6a422aa6467f00864c01f4 (table default.Table_c3fec71fedac4b4c9864b18ea6af10b9 [id=382fc98d4c614a9fb79d50a9d630b36d]) successfully deleted
I20250115 14:52:04.407953 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 574) took 2128 ms. Trace:
I20250115 14:52:04.408088 1167 rpcz_store.cc:276] 0115 14:52:02.279160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.284873 (+ 5713us) service_pool.cc:225] Handling call
0115 14:52:04.407854 (+2122981us) ts_tablet_manager.cc:1145] Deleting tablet e6978223c1984a38bec5f2d06a9cf457
0115 14:52:04.407863 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.407939 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2084099}
W20250115 14:52:04.408586 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e6978223c1984a38bec5f2d06a9cf457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6978223c1984a38bec5f2d06a9cf457 already in progress: creating tablet
I20250115 14:52:04.408977 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 573) took 2130 ms. Trace:
I20250115 14:52:04.409091 1167 rpcz_store.cc:276] 0115 14:52:02.279001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.288032 (+ 9031us) service_pool.cc:225] Handling call
0115 14:52:04.408874 (+2120842us) ts_tablet_manager.cc:1145] Deleting tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a
0115 14:52:04.408882 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.408952 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2084989}
W20250115 14:52:04.409489 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a already in progress: creating tablet
I20250115 14:52:04.409555 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2edd5570ca604ac6bd85686e0e708187 (DEFAULT_TABLE table=default.Table_5c73eff7b16a4a11ad47cd4c32a6fd0a [id=c1dc41883d4448d882e8acc833ebc260]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.410159 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 575) took 1842 ms. Trace:
I20250115 14:52:04.410279 1167 rpcz_store.cc:276] 0115 14:52:02.567785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.567849 (+ 64us) service_pool.cc:225] Handling call
0115 14:52:04.410057 (+1842208us) ts_tablet_manager.cc:1145] Deleting tablet 7d612b424bea408cb6691c7c200cf3a6
0115 14:52:04.410066 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.410144 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1841977}
I20250115 14:52:04.413064 2117 leader_election.cc:290] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:04.414585 710 leader_election.cc:343] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 496dec75500347699625a755dceed2bb
I20250115 14:52:04.421190 2117 consensus_queue.cc:1035] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:04.422122 750 tablet_service.cc:1467] Processing CreateTablet for tablet 6b28a180aa0e4ef2b62890233cd512a4 (DEFAULT_TABLE table=default.Table_00e45525259140518db638102c114d02 [id=aa031221a7034b62a85c8914d6872ba7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.422533 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b28a180aa0e4ef2b62890233cd512a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.405189 1751 ts_tablet_manager.cc:1918] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:04.428026 1751 log.cc:1198] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/7e3ec000089a47a2930dec382192cd6f
I20250115 14:52:04.428450 1751 ts_tablet_manager.cc:1939] T 7e3ec000089a47a2930dec382192cd6f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:04.428961 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 (DEFAULT_TABLE table=default.Table_d3d1424de57f42b89ddb3d6f1cbff283 [id=8c74fcd221c3405a97218d0ff2c91356]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.429097 743 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 (DEFAULT_TABLE table=default.Table_d3d1424de57f42b89ddb3d6f1cbff283 [id=8c74fcd221c3405a97218d0ff2c91356]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.429764 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.430578 745 tablet_service.cc:1467] Processing CreateTablet for tablet 919ab6ec6f3745e082aa441dfebc235e (DEFAULT_TABLE table=default.Table_9b06f855e23e47cbb239075f101b585c [id=d6f2abb6e8894bf58f3aa4876cc38ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.411408 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2edd5570ca604ac6bd85686e0e708187. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:04.431166 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7d612b424bea408cb6691c7c200cf3a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d612b424bea408cb6691c7c200cf3a6 already in progress: creating tablet
I20250115 14:52:04.430912 741 tablet_service.cc:1467] Processing CreateTablet for tablet 821bff4624de48d59005368b8f8fefc4 (DEFAULT_TABLE table=default.Table_4407280ad4d5424b95ea7687acf680ee [id=72f4c90627ff48889f0424f4653e5a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.431535 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 585) took 1640 ms. Trace:
I20250115 14:52:04.431609 1167 rpcz_store.cc:276] 0115 14:52:02.790701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.790754 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:04.431410 (+1640656us) ts_tablet_manager.cc:1145] Deleting tablet 6942d102069d4c33b23fa2c0b29f94a6
0115 14:52:04.431430 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.431517 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1640492}
W20250115 14:52:04.431988 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6942d102069d4c33b23fa2c0b29f94a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6942d102069d4c33b23fa2c0b29f94a6 already in progress: creating tablet
I20250115 14:52:04.434290 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 587) took 1643 ms. Trace:
I20250115 14:52:04.434360 1167 rpcz_store.cc:276] 0115 14:52:02.790889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.790948 (+ 59us) service_pool.cc:225] Handling call
0115 14:52:04.434192 (+1643244us) ts_tablet_manager.cc:1145] Deleting tablet c574dba6b2a04915bdd871de4dff4df0
0115 14:52:04.434201 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.434275 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643106}
W20250115 14:52:04.435269 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c574dba6b2a04915bdd871de4dff4df0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c574dba6b2a04915bdd871de4dff4df0 already in progress: creating tablet
I20250115 14:52:04.415279 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496dec75500347699625a755dceed2bb" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:04.435863 911 raft_consensus.cc:3054] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.438571 911 raft_consensus.cc:2463] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:04.438690 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 586) took 1647 ms. Trace:
I20250115 14:52:04.438760 1167 rpcz_store.cc:276] 0115 14:52:02.790810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.791306 (+ 496us) service_pool.cc:225] Handling call
0115 14:52:04.438596 (+1647290us) ts_tablet_manager.cc:1145] Deleting tablet e244689cef6844e2935becd265eecad0
0115 14:52:04.438605 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.438676 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1647165}
W20250115 14:52:04.439111 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e244689cef6844e2935becd265eecad0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e244689cef6844e2935becd265eecad0 already in progress: creating tablet
I20250115 14:52:04.440542 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 588) took 1649 ms. Trace:
I20250115 14:52:04.421066 1168 tablet_bootstrap.cc:492] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:04.442426 1167 rpcz_store.cc:276] 0115 14:52:02.790993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.791457 (+ 464us) service_pool.cc:225] Handling call
0115 14:52:04.440447 (+1648990us) ts_tablet_manager.cc:1145] Deleting tablet b76d1bbf90c6416ca60805c9d8053bd4
0115 14:52:04.440456 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.440526 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1648877}
I20250115 14:52:04.442548 1168 ts_tablet_manager.cc:1397] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.001s
I20250115 14:52:04.419219 880 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 (DEFAULT_TABLE table=default.Table_d3d1424de57f42b89ddb3d6f1cbff283 [id=8c74fcd221c3405a97218d0ff2c91356]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:04.442804 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b76d1bbf90c6416ca60805c9d8053bd4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b76d1bbf90c6416ca60805c9d8053bd4 already in progress: creating tablet
I20250115 14:52:04.443316 710 leader_election.cc:304] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:04.443456 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 584) took 1652 ms. Trace:
I20250115 14:52:04.443838 2117 raft_consensus.cc:2798] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:04.443866 1167 rpcz_store.cc:276] 0115 14:52:02.790595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.791989 (+ 1394us) service_pool.cc:225] Handling call
0115 14:52:04.443363 (+1651374us) ts_tablet_manager.cc:1145] Deleting tablet 5b85d63974ab418eb1921dd605fbf2b0
0115 14:52:04.443372 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.443443 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651096}
W20250115 14:52:04.444408 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 5b85d63974ab418eb1921dd605fbf2b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b85d63974ab418eb1921dd605fbf2b0 already in progress: creating tablet
I20250115 14:52:04.444470 2117 raft_consensus.cc:695] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:04.444886 2117 consensus_queue.cc:237] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.445091 1167 tablet_replica.cc:331] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:04.445297 1167 raft_consensus.cc:2238] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:04.445598 1167 pending_rounds.cc:62] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:04.446060 1167 pending_rounds.cc:69] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558351713968128 op_type: NO_OP noop_request { }
I20250115 14:52:04.446214 1167 raft_consensus.cc:2883] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:04.446561 749 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4238c63c4c42aba30f906a87c644a2 (DEFAULT_TABLE table=default.Table_0b2109f1c1814a63a365822b2102e61e [id=96b3610ffe9647e2ae452779d96fef96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.446776 1167 raft_consensus.cc:2267] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.447937 1167 ts_tablet_manager.cc:1905] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:04.449568 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5df0250d08214a9783c392071126ed4a consensus state has no leader
W20250115 14:52:04.449921 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 171643c40c9b43ffb823f7c33789284d consensus state has no leader
I20250115 14:52:04.451184 455 catalog_manager.cc:5526] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d7ecece2ab49e1be7079216517d13b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d2d7ecece2ab49e1be7079216517d13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:04.452021 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a26e212086b1478cb159097102934dd1" } modify_external_catalogs: true
I20250115 14:52:04.452209 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a26e212086b1478cb159097102934dd1" } modify_external_catalogs: true
W20250115 14:52:04.453025 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b36ca2df86b84a509fba08301f46a04f consensus state has no leader
I20250115 14:52:04.453593 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502] alter complete (version 2)
I20250115 14:52:04.454314 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_44bc726782ff4d8d8a341acf255a4dd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.454932 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.455682 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_69ed5f68465549b38e7e27bf0d75eb2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.456008 2059 log.cc:826] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: Log is configured to *not* fsync() on all Append() calls
I20250115 14:52:04.457656 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 3e4b8b77c11a4582b0d6b85f4c49aa82 (DEFAULT_TABLE table=default.Table_24d51c1e7d8d4fdd83f72d6094af1d6b [id=8e3ab65e00e54c92a2e1bb98163c77ac]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:04.458441 2093 log.cc:926] Time spent T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: Append to log took a long time: real 0.201s user 0.001s sys 0.000s
I20250115 14:52:04.458807 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_80b26a41f06c4b69a4cf5ef46f85fc8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.458916 2122 log.cc:926] Time spent T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
W20250115 14:52:04.459244 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80b26a41f06c4b69a4cf5ef46f85fc8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:04.459244 2118 consensus_queue.cc:1159] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d2d7ecece2ab49e1be7079216517d13b" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250115 14:52:04.459991 1166 ts_tablet_manager.cc:1905] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.460541 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 282d988bc41443c381bb0ba69b618f5d from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:04.461503 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44bc726782ff4d8d8a341acf255a4dd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.462402 1168 raft_consensus.cc:357] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.462476 2117 consensus_queue.cc:1035] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:04.462473 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_afd9df7559aa485789c8e7bd5d43c3ef" } modify_external_catalogs: true
I20250115 14:52:04.462576 1168 raft_consensus.cc:383] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.462622 1168 raft_consensus.cc:738] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.462630 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_afd9df7559aa485789c8e7bd5d43c3ef" } modify_external_catalogs: true
I20250115 14:52:04.462810 1168 consensus_queue.cc:260] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.463256 1168 ts_tablet_manager.cc:1428] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250115 14:52:04.463524 1168 tablet_bootstrap.cc:492] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:04.463634 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 856533ed254a49b4a461ed1ad3eb405e consensus state has no leader
I20250115 14:52:04.463971 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a99495ed1c544a6f825fa19b1e44009f" } modify_external_catalogs: true
I20250115 14:52:04.464133 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a99495ed1c544a6f825fa19b1e44009f" } modify_external_catalogs: true
I20250115 14:52:04.464315 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 919ab6ec6f3745e082aa441dfebc235e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.465054 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a4238c63c4c42aba30f906a87c644a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.465584 1167 ts_tablet_manager.cc:1918] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:04.465677 1167 log.cc:1198] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/6f07ef30cac2470d9b2f428338dd37a3
W20250115 14:52:04.466295 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69ed5f68465549b38e7e27bf0d75eb2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.466800 1167 ts_tablet_manager.cc:1939] T 6f07ef30cac2470d9b2f428338dd37a3 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:04.467080 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 590) took 1654 ms. Trace:
I20250115 14:52:04.467231 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a915e0c0a30940ed8a6233f117e2a560
I20250115 14:52:04.467150 1167 rpcz_store.cc:276] 0115 14:52:02.813009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.820402 (+ 7393us) service_pool.cc:225] Handling call
0115 14:52:04.445052 (+1624650us) ts_tablet_manager.cc:1145] Deleting tablet 6f07ef30cac2470d9b2f428338dd37a3
0115 14:52:04.445060 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.451380 (+ 6320us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.465500 (+ 14120us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.467061 (+ 1561us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1624470}
I20250115 14:52:04.467525 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 6f07ef30cac2470d9b2f428338dd37a3 (table default.Table_17b896568dc34b1aaf2b3e197d03fdc4 [id=b9d32cae6ebd42cea68dc15f454318ab]) successfully deleted
I20250115 14:52:04.467657 452 catalog_manager.cc:6596] ReplaceTablet: tablet a915e0c0a30940ed8a6233f117e2a560 deleted and replaced by tablet f5c1f0e0b6ee4448b9ba2f78d0117197
I20250115 14:52:04.467919 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6b196b51d11442e5be9d5a3a2f6f3484" } modify_external_catalogs: true
I20250115 14:52:04.468077 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6b196b51d11442e5be9d5a3a2f6f3484" } modify_external_catalogs: true
I20250115 14:52:04.468338 1167 tablet_replica.cc:331] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:04.468379 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet a915e0c0a30940ed8a6233f117e2a560 with delete_type TABLET_DATA_DELETED (replaced by tablet f5c1f0e0b6ee4448b9ba2f78d0117197) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.468502 1167 raft_consensus.cc:2238] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:04.468837 1167 raft_consensus.cc:2267] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.468869 889 tablet_service.cc:1514] Processing DeleteTablet for tablet a915e0c0a30940ed8a6233f117e2a560 with delete_type TABLET_DATA_DELETED (replaced by tablet f5c1f0e0b6ee4448b9ba2f78d0117197) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.469321 1167 ts_tablet_manager.cc:1905] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.470593 750 tablet_service.cc:1514] Processing DeleteTablet for tablet a915e0c0a30940ed8a6233f117e2a560 with delete_type TABLET_DATA_DELETED (replaced by tablet f5c1f0e0b6ee4448b9ba2f78d0117197) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.473743 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_29e435c06be34d20b3fe6ab2745835d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.474177 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29e435c06be34d20b3fe6ab2745835d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.474154 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ad2b7350294375bd30fb4b46906491 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.475033 1168 tablet_bootstrap.cc:654] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.488128 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 821bff4624de48d59005368b8f8fefc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.491070 1166 ts_tablet_manager.cc:1918] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:04.491192 1166 log.cc:1198] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/23cba5cc51d7414a9352caedcc362c9c
I20250115 14:52:04.491497 1166 ts_tablet_manager.cc:1939] T 23cba5cc51d7414a9352caedcc362c9c P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:04.492224 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 23cba5cc51d7414a9352caedcc362c9c (table default.Table_27cd4d793c8f4e77aa531c06dfa74a2d [id=a6994f260e1a4eada78d8c9e9fcf334e]) successfully deleted
I20250115 14:52:04.492488 1166 tablet_replica.cc:331] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:04.492671 1166 raft_consensus.cc:2238] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:04.492695 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9853dcdfee5f43de84798cfa0bef674e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9853dcdfee5f43de84798cfa0bef674e already in progress: creating tablet
I20250115 14:52:04.492848 1166 raft_consensus.cc:2267] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:04.493112 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa already in progress: creating tablet
W20250115 14:52:04.493362 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 342bf8d4dc4244f982b1b63aaae0fb45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 342bf8d4dc4244f982b1b63aaae0fb45 already in progress: creating tablet
I20250115 14:52:04.493376 1166 ts_tablet_manager.cc:1905] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:04.498654 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 655489ad9bd34be0a8283a295f583092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 655489ad9bd34be0a8283a295f583092 already in progress: creating tablet
I20250115 14:52:04.502647 2131 tablet.cc:1721] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:04.504582 1168 tablet_bootstrap.cc:492] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:04.504789 1168 ts_tablet_manager.cc:1397] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.001s
W20250115 14:52:04.506778 848 consensus_peers.cc:487] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3a7a487a42949bd9641c38d6c1fcfd3. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.503542 1166 ts_tablet_manager.cc:1918] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:04.507068 1166 log.cc:1198] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/4821224ac8f1472d8b1e8afc06a118a3
I20250115 14:52:04.507359 1168 raft_consensus.cc:357] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.507611 1168 raft_consensus.cc:383] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.507704 1168 raft_consensus.cc:738] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.507934 1168 consensus_queue.cc:260] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.508579 1168 ts_tablet_manager.cc:1428] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250115 14:52:04.508890 1168 tablet_bootstrap.cc:492] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:04.512768 848 consensus_peers.cc:487] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9e5bc6131944fcab9c550caac4f8d85. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.513249 1166 ts_tablet_manager.cc:1939] T 4821224ac8f1472d8b1e8afc06a118a3 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:52:04.514626 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2e4c2707915e4e97ad6c66cb249c6d9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4c2707915e4e97ad6c66cb249c6d9e already in progress: creating tablet
W20250115 14:52:04.515692 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c1a77d07b1c946b18e0769a37d8a50a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1a77d07b1c946b18e0769a37d8a50a1 already in progress: creating tablet
W20250115 14:52:04.517567 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e180e37a1d5d46d78380323662b80f85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e180e37a1d5d46d78380323662b80f85 already in progress: creating tablet
I20250115 14:52:04.517881 1166 tablet_replica.cc:331] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:04.518106 1166 raft_consensus.cc:2238] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.518282 1166 raft_consensus.cc:2267] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.519814 1166 ts_tablet_manager.cc:1905] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.529001 1164 tablet_bootstrap.cc:492] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:04.529186 1164 ts_tablet_manager.cc:1397] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.340s user 0.005s sys 0.007s
I20250115 14:52:04.529898 1164 raft_consensus.cc:357] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.530210 1164 raft_consensus.cc:383] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.530308 1164 raft_consensus.cc:738] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.530566 1164 consensus_queue.cc:260] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.531234 1164 ts_tablet_manager.cc:1428] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:04.531554 1164 tablet_bootstrap.cc:492] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:04.532908 1164 tablet_bootstrap.cc:654] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.542317 1876 raft_consensus.cc:491] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.542455 1876 raft_consensus.cc:513] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.543056 1876 leader_election.cc:290] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.543540 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f793828f2ca49639c57849afc61ce9e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:04.544061 983 leader_election.cc:343] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.552717 1168 tablet_bootstrap.cc:654] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.555256 1167 ts_tablet_manager.cc:1918] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:04.561156 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 821bff4624de48d59005368b8f8fefc4 (DEFAULT_TABLE table=default.Table_4407280ad4d5424b95ea7687acf680ee [id=72f4c90627ff48889f0424f4653e5a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.561628 880 tablet_service.cc:1467] Processing CreateTablet for tablet 821bff4624de48d59005368b8f8fefc4 (DEFAULT_TABLE table=default.Table_4407280ad4d5424b95ea7687acf680ee [id=72f4c90627ff48889f0424f4653e5a4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.562848 889 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc5e4c7ec1f426dadbafe85d5eebdc9 (DEFAULT_TABLE table=default.Table_15364f05c206457da9c472858167e5dd [id=c0df14a2bb204902a943cecca00298d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.568482 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc5e4c7ec1f426dadbafe85d5eebdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.569532 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 398ac8af619c427287b3964443173864
I20250115 14:52:04.570744 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 282d988bc41443c381bb0ba69b618f5d
I20250115 14:52:04.571125 449 catalog_manager.cc:6596] ReplaceTablet: tablet 282d988bc41443c381bb0ba69b618f5d deleted and replaced by tablet 567365040b6d4d6eba04112c8d4c3dfb
I20250115 14:52:04.572240 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 105b6148ae944e5c906e07d987177cc5
I20250115 14:52:04.573020 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 74d00bcee22d40769102eacaf728480d
I20250115 14:52:04.573540 448 master_service.cc:946] ReplaceTablet: received request to replace tablet ca5cc55b50ee4953800f10efe0ffb546 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:04.574086 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 48b15de9bf8943cdb9e5abd943484993 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:04.574666 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 821bff4624de48d59005368b8f8fefc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.575743 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9494c0c57be8499a883888d067c7db77
I20250115 14:52:04.577711 2013 raft_consensus.cc:491] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.577836 2013 raft_consensus.cc:513] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.578382 2013 leader_election.cc:290] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:04.581727 1167 log.cc:1198] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d97c2f443ab040f9bc1924f45379165e
I20250115 14:52:04.581918 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 282d988bc41443c381bb0ba69b618f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 567365040b6d4d6eba04112c8d4c3dfb) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.582165 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 105b6148ae944e5c906e07d987177cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.582226 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d00bcee22d40769102eacaf728480d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.582281 1167 ts_tablet_manager.cc:1939] T d97c2f443ab040f9bc1924f45379165e P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:04.582441 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 398ac8af619c427287b3964443173864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.582657 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 9494c0c57be8499a883888d067c7db77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.583389 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 282d988bc41443c381bb0ba69b618f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 567365040b6d4d6eba04112c8d4c3dfb) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.583606 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d00bcee22d40769102eacaf728480d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.583698 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 9494c0c57be8499a883888d067c7db77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.583801 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 398ac8af619c427287b3964443173864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.584487 1168 tablet_bootstrap.cc:492] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:04.584645 1168 ts_tablet_manager.cc:1397] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.002s
I20250115 14:52:04.584666 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 105b6148ae944e5c906e07d987177cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.585572 1168 raft_consensus.cc:357] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.585893 1168 raft_consensus.cc:383] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.585984 1168 raft_consensus.cc:738] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.563975 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc5e4c7ec1f426dadbafe85d5eebdc9 (DEFAULT_TABLE table=default.Table_15364f05c206457da9c472858167e5dd [id=c0df14a2bb204902a943cecca00298d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.587033 1168 consensus_queue.cc:260] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.590281 1168 ts_tablet_manager.cc:1428] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:52:04.590611 1168 tablet_bootstrap.cc:492] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:04.591312 1876 raft_consensus.cc:491] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.591511 1876 raft_consensus.cc:513] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.592098 1876 leader_election.cc:290] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.592206 1168 tablet_bootstrap.cc:654] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.593230 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01291d41a2b345b4bfd00e6be0fb0129" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:04.593700 983 leader_election.cc:343] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.593300 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 589) took 1780 ms. Trace:
I20250115 14:52:04.595098 1167 rpcz_store.cc:276] 0115 14:52:02.812899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.821696 (+ 8797us) service_pool.cc:225] Handling call
0115 14:52:04.468279 (+1646583us) ts_tablet_manager.cc:1145] Deleting tablet d97c2f443ab040f9bc1924f45379165e
0115 14:52:04.468305 (+ 26us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.550472 (+ 82167us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.555164 (+ 4692us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.593283 (+ 38119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646421}
I20250115 14:52:04.595585 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d97c2f443ab040f9bc1924f45379165e (table default.Table_121cfa7da22441d581d821791e428785 [id=470df20dd8bd47f4acfa2d40f2e034bd]) successfully deleted
I20250115 14:52:04.595664 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 591) took 1767 ms. Trace:
I20250115 14:52:04.595824 1167 rpcz_store.cc:276] 0115 14:52:02.828179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.830652 (+ 2473us) service_pool.cc:225] Handling call
0115 14:52:04.595552 (+1764900us) ts_tablet_manager.cc:1145] Deleting tablet 72a97d102c174bc3a5e3a3c5762033b3
0115 14:52:04.595562 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.595648 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1757004}
W20250115 14:52:04.596249 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 72a97d102c174bc3a5e3a3c5762033b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72a97d102c174bc3a5e3a3c5762033b3 already in progress: creating tablet
I20250115 14:52:04.596321 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 592) took 1718 ms. Trace:
I20250115 14:52:04.596477 1167 rpcz_store.cc:276] 0115 14:52:02.877663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.877763 (+ 100us) service_pool.cc:225] Handling call
0115 14:52:04.596210 (+1718447us) ts_tablet_manager.cc:1145] Deleting tablet 2560d42c15c64b4c9c0cfe95706b4d9e
0115 14:52:04.596218 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.596307 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1717329}
I20250115 14:52:04.596982 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 593) took 1717 ms. Trace:
I20250115 14:52:04.597096 1167 rpcz_store.cc:276] 0115 14:52:02.879058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.879107 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:04.596894 (+1717787us) ts_tablet_manager.cc:1145] Deleting tablet 1b2d04209fd34918b83978325a5e1df8
0115 14:52:04.596902 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.596968 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1717612}
I20250115 14:52:04.597529 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 595) took 1697 ms. Trace:
I20250115 14:52:04.597642 1167 rpcz_store.cc:276] 0115 14:52:02.900264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.900352 (+ 88us) service_pool.cc:225] Handling call
0115 14:52:04.597442 (+1697090us) ts_tablet_manager.cc:1145] Deleting tablet 5265d5eeac83481ca97801e3475886da
0115 14:52:04.597450 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.597515 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1693796}
I20250115 14:52:04.598533 2013 consensus_queue.cc:1035] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:04.598201 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd41d42331494e9c1b021816d0287e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:04.598930 1048 raft_consensus.cc:2463] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:04.599486 847 leader_election.cc:304] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:04.599790 2013 raft_consensus.cc:2798] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:04.599857 2013 raft_consensus.cc:491] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:04.599910 2013 raft_consensus.cc:3054] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.602345 2013 raft_consensus.cc:513] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.602901 2013 leader_election.cc:290] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:04.602980 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2560d42c15c64b4c9c0cfe95706b4d9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2560d42c15c64b4c9c0cfe95706b4d9e already in progress: creating tablet
W20250115 14:52:04.603247 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 1b2d04209fd34918b83978325a5e1df8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b2d04209fd34918b83978325a5e1df8 already in progress: creating tablet
I20250115 14:52:04.604921 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.606945 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 821bff4624de48d59005368b8f8fefc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.608409 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 648) took 1163 ms. Trace:
I20250115 14:52:04.608484 1751 rpcz_store.cc:276] 0115 14:52:03.444971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.445019 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:04.337542 (+892523us) ts_tablet_manager.cc:1145] Deleting tablet 7e3ec000089a47a2930dec382192cd6f
0115 14:52:04.337551 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.372343 (+ 34792us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.404807 (+ 32464us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.608395 (+203588us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":892359}
I20250115 14:52:04.608772 1751 tablet_replica.cc:331] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:04.608933 1751 raft_consensus.cc:2238] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.609052 1751 raft_consensus.cc:2267] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.609499 1751 ts_tablet_manager.cc:1905] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.611276 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd41d42331494e9c1b021816d0287e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:04.611425 1048 raft_consensus.cc:3054] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.614094 1048 raft_consensus.cc:2463] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:04.615115 1751 ts_tablet_manager.cc:1918] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.615213 1751 log.cc:1198] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/caed629878594954842dfc6a40cc8b6d
I20250115 14:52:04.615201 847 leader_election.cc:304] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:04.615482 1751 ts_tablet_manager.cc:1939] T caed629878594954842dfc6a40cc8b6d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:04.616035 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 650) took 1137 ms. Trace:
I20250115 14:52:04.616165 1751 rpcz_store.cc:276] 0115 14:52:03.478361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.484792 (+ 6431us) service_pool.cc:225] Handling call
0115 14:52:04.608726 (+1123934us) ts_tablet_manager.cc:1145] Deleting tablet caed629878594954842dfc6a40cc8b6d
0115 14:52:04.608741 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.612379 (+ 3638us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.615044 (+ 2665us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.616020 (+ 976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1123714}
I20250115 14:52:04.616393 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]) successfully deleted
I20250115 14:52:04.616803 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 652) took 1094 ms. Trace:
I20250115 14:52:04.616928 1751 rpcz_store.cc:276] 0115 14:52:03.522671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.522736 (+ 65us) service_pool.cc:225] Handling call
0115 14:52:04.616599 (+1093863us) ts_tablet_manager.cc:1145] Deleting tablet 02ce16fea4f9469c9fc06fa24f6554be
0115 14:52:04.616608 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.616787 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1093640}
I20250115 14:52:04.617338 1751 tablet_replica.cc:331] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:04.617641 1751 raft_consensus.cc:2238] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.617717 1843 raft_consensus.cc:2798] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:04.617897 1751 raft_consensus.cc:2267] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.618721 1751 ts_tablet_manager.cc:1905] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:04.619998 848 consensus_peers.cc:487] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.622285 1048 raft_consensus.cc:1270] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:04.622884 2083 consensus_queue.cc:1035] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:04.623821 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet caed629878594954842dfc6a40cc8b6d (table default.Table_544d606cf9af4accb670d91d7d717948 [id=b6aeb933c5e64e88a6d794a98650acbe]) successfully deleted
W20250115 14:52:04.624132 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 02ce16fea4f9469c9fc06fa24f6554be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02ce16fea4f9469c9fc06fa24f6554be already in progress: creating tablet
I20250115 14:52:04.626041 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc5e4c7ec1f426dadbafe85d5eebdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.634037 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e4b8b77c11a4582b0d6b85f4c49aa82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.636420 1167 tablet_replica.cc:331] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
W20250115 14:52:04.636376 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 5265d5eeac83481ca97801e3475886da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5265d5eeac83481ca97801e3475886da already in progress: creating tablet
I20250115 14:52:04.636693 1167 raft_consensus.cc:2238] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.636889 1167 raft_consensus.cc:2267] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.637539 1167 ts_tablet_manager.cc:1905] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.645676 1751 ts_tablet_manager.cc:1918] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.646342 1751 log.cc:1198] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/be63bae96c834a8bbbb1174c027935a2
I20250115 14:52:04.648082 1751 ts_tablet_manager.cc:1939] T be63bae96c834a8bbbb1174c027935a2 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:04.649854 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 655) took 1062 ms. Trace:
I20250115 14:52:04.650264 1751 rpcz_store.cc:276] 0115 14:52:03.587885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.588180 (+ 295us) service_pool.cc:225] Handling call
0115 14:52:04.617273 (+1029093us) ts_tablet_manager.cc:1145] Deleting tablet be63bae96c834a8bbbb1174c027935a2
0115 14:52:04.617281 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.626103 (+ 8822us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.645170 (+ 19067us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.649834 (+ 4664us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028893}
I20250115 14:52:04.535539 1166 ts_tablet_manager.cc:1918] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.651628 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet be63bae96c834a8bbbb1174c027935a2 (table default.Table_705745b616194094bb03915f2f651122 [id=b2c6ee1ec113494cac7d1bf11a928714]) successfully deleted
I20250115 14:52:04.651750 1166 log.cc:1198] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e198b4e1a65b4704bf2bb8f7d5987385
I20250115 14:52:04.651933 1168 tablet_bootstrap.cc:492] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:04.652060 1166 ts_tablet_manager.cc:1939] T e198b4e1a65b4704bf2bb8f7d5987385 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:04.652102 1168 ts_tablet_manager.cc:1397] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20250115 14:52:04.543699 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f793828f2ca49639c57849afc61ce9e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:04.572839 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 105b6148ae944e5c906e07d987177cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.573546 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d00bcee22d40769102eacaf728480d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.576284 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 9494c0c57be8499a883888d067c7db77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.583925 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd41d42331494e9c1b021816d0287e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:04.652901 1168 raft_consensus.cc:357] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.653082 1168 raft_consensus.cc:383] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.653131 1168 raft_consensus.cc:738] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
W20250115 14:52:04.653193 984 leader_election.cc:343] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.653290 984 leader_election.cc:304] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.653318 1168 consensus_queue.cc:260] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.652985 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 656) took 1043 ms. Trace:
I20250115 14:52:04.653788 1168 ts_tablet_manager.cc:1428] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:04.654022 1168 tablet_bootstrap.cc:492] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:04.655469 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb47732109e46d68e5af02ed52cb6d1 (DEFAULT_TABLE table=default.Table_9591925055134e48ad3592198934e708 [id=85daeb1cc82944aa83686987624e038c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.656030 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cb47732109e46d68e5af02ed52cb6d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.656462 1168 tablet_bootstrap.cc:654] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.656569 1166 tablet_replica.cc:331] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:04.656751 1166 raft_consensus.cc:2238] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:04.656968 1166 pending_rounds.cc:62] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:04.657008 1166 pending_rounds.cc:69] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558359329275904 op_type: NO_OP noop_request { }
I20250115 14:52:04.657121 1166 raft_consensus.cc:2883] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:04.657188 1166 raft_consensus.cc:2267] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.657516 1843 raft_consensus.cc:695] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
W20250115 14:52:04.657966 848 leader_election.cc:343] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.657939 1843 consensus_queue.cc:237] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.658445 751 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb47732109e46d68e5af02ed52cb6d1 (DEFAULT_TABLE table=default.Table_9591925055134e48ad3592198934e708 [id=85daeb1cc82944aa83686987624e038c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.658597 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e198b4e1a65b4704bf2bb8f7d5987385 (table default.Table_8a0954ff11744933ba4752116830cdd7 [id=a3bf28673dac4ddab579709d6fd8885d]) successfully deleted
I20250115 14:52:04.658764 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cb47732109e46d68e5af02ed52cb6d1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:04.658828 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 11fddbd9ba14443a91865387e1d1c238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11fddbd9ba14443a91865387e1d1c238 already in progress: creating tablet
W20250115 14:52:04.659062 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ed777d7496514ec5986a7e6f53d532b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed777d7496514ec5986a7e6f53d532b1 already in progress: creating tablet
I20250115 14:52:04.653582 1751 rpcz_store.cc:276] 0115 14:52:03.609539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.613623 (+ 4084us) service_pool.cc:225] Handling call
0115 14:52:04.652353 (+1038730us) ts_tablet_manager.cc:1145] Deleting tablet d9ce65604c294021af93632ed073f627
0115 14:52:04.652459 (+ 106us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.652865 (+ 406us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038411}
W20250115 14:52:04.662840 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb already in progress: creating tablet
I20250115 14:52:04.662902 879 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb47732109e46d68e5af02ed52cb6d1 (DEFAULT_TABLE table=default.Table_9591925055134e48ad3592198934e708 [id=85daeb1cc82944aa83686987624e038c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.663146 1751 tablet_replica.cc:331] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:04.663399 1751 raft_consensus.cc:2238] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.663614 1751 raft_consensus.cc:2267] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.664278 1166 ts_tablet_manager.cc:1905] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.664384 1751 ts_tablet_manager.cc:1905] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.667393 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e215d4b1143c4ea49dde3acc698b7e60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.667960 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e215d4b1143c4ea49dde3acc698b7e60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.584833 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 282d988bc41443c381bb0ba69b618f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 567365040b6d4d6eba04112c8d4c3dfb) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.588121 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 398ac8af619c427287b3964443173864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.592653 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01291d41a2b345b4bfd00e6be0fb0129" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:04.614468 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5dd41d42331494e9c1b021816d0287e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:04.670576 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cb47732109e46d68e5af02ed52cb6d1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:04.670601 848 leader_election.cc:343] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.648041 741 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc5e4c7ec1f426dadbafe85d5eebdc9 (DEFAULT_TABLE table=default.Table_15364f05c206457da9c472858167e5dd [id=c0df14a2bb204902a943cecca00298d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:04.671123 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc5e4c7ec1f426dadbafe85d5eebdc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:04.674621 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_98190516662f4514bd887e05fe683703" } modify_external_catalogs: true
I20250115 14:52:04.674806 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_98190516662f4514bd887e05fe683703" } modify_external_catalogs: true
I20250115 14:52:04.675563 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_01f73e93e1ba462097edfe0e8aaba189" } new_table_name: "default.Table_e79474d461c04307bd06aac8b3a66328" modify_external_catalogs: true
I20250115 14:52:04.675760 1166 ts_tablet_manager.cc:1918] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:04.675854 1166 log.cc:1198] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/496dec75500347699625a755dceed2bb
I20250115 14:52:04.676177 1166 ts_tablet_manager.cc:1939] T 496dec75500347699625a755dceed2bb P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:04.677165 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 496dec75500347699625a755dceed2bb (table default.Table_cb0308c238d7479dbcc041fe3d6e7eff [id=983e866df38543b0b2502165df44b94c]) successfully deleted
I20250115 14:52:04.677860 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_67307f5008004a1fa8f71bf061edd60a" } new_table_name: "default.Table_ac31746e6f2f4855b46e06e0f7d95a16" modify_external_catalogs: true
I20250115 14:52:04.679196 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:45653: OK, instance_id { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" instance_seqno: 1736952722612014 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250115 14:52:04.679421 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250115 14:52:04.680032 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet fbf94077e0e343b8938b081102dfd2af (table default.Table_bbd25ad975704fbcae339bf950fd1052 [id=00a89ba1b93f4132aae8fe6e748df77e]). Expected version 2 got 0
I20250115 14:52:04.680220 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:04.681062 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9c44f30ee706467f8ae6014dd1b51473" } new_table_name: "default.Table_7db2f6193de84d23a01b711b7de04815" modify_external_catalogs: true
I20250115 14:52:04.682117 1167 ts_tablet_manager.cc:1918] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.682073 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_087db70e94b0414c83610ae7e656c9d0" } new_table_name: "default.Table_73d45488e7de41b8ba166c493c6ce4d4" modify_external_catalogs: true
I20250115 14:52:04.682206 1167 log.cc:1198] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/23cba5cc51d7414a9352caedcc362c9c
I20250115 14:52:04.683169 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3f9ece922e604339970891650343338a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.683642 1168 tablet_bootstrap.cc:492] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
W20250115 14:52:04.683640 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f9ece922e604339970891650343338a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.683768 1168 ts_tablet_manager.cc:1397] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20250115 14:52:04.684266 1166 tablet_replica.cc:331] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:04.684473 1166 raft_consensus.cc:2238] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.684492 1168 raft_consensus.cc:357] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.684635 1166 raft_consensus.cc:2267] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.684654 1168 raft_consensus.cc:383] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.684710 1168 raft_consensus.cc:738] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.687067 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:04.687225 1166 ts_tablet_manager.cc:1905] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.687546 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 48b15de9bf8943cdb9e5abd943484993
I20250115 14:52:04.688016 454 catalog_manager.cc:6596] ReplaceTablet: tablet 48b15de9bf8943cdb9e5abd943484993 deleted and replaced by tablet 1de35551e26643b3b1e570ab8b2f6f56
I20250115 14:52:04.688215 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b15de9bf8943cdb9e5abd943484993 with delete_type TABLET_DATA_DELETED (replaced by tablet 1de35551e26643b3b1e570ab8b2f6f56) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.688472 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_468c64853fef4ebfa521100c23cb2620"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.688920 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_468c64853fef4ebfa521100c23cb2620 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.690191 447 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:04.690337 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 5acfe27cdf904b92b426d55d78687629
I20250115 14:52:04.690699 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_28ceb002e729415a92c24a593b1ba77a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.691151 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28ceb002e729415a92c24a593b1ba77a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.691599 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ba297fd1bf8c47dabb7b1dab458cf8df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.692029 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba297fd1bf8c47dabb7b1dab458cf8df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.692207 749 tablet_service.cc:1514] Processing DeleteTablet for tablet d195be2d543f4a8693045a024912c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.692399 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 496dec75500347699625a755dceed2bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.693840 1168 consensus_queue.cc:260] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.694273 1167 ts_tablet_manager.cc:1939] T 23cba5cc51d7414a9352caedcc362c9c P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:04.695365 1168 ts_tablet_manager.cc:1428] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250115 14:52:04.695612 1168 tablet_bootstrap.cc:492] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:04.697806 848 consensus_peers.cc:487] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.699689 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:04.700160 452 catalog_manager.cc:5526] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:04.701085 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]). Expected version 1 got 0
I20250115 14:52:04.703640 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 596) took 1792 ms. Trace:
W20250115 14:52:04.703825 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dcf442cc39c64f7ba3f78c083cd17b94 has no consensus state
W20250115 14:52:04.704195 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec2a8000f5474f62aae52c8e88e45276 consensus state has no leader
I20250115 14:52:04.704237 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a69fe39cbd8f4a628213badbaafff14a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.704249 1167 rpcz_store.cc:276] 0115 14:52:02.911045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.911470 (+ 425us) service_pool.cc:225] Handling call
0115 14:52:04.636365 (+1724895us) ts_tablet_manager.cc:1145] Deleting tablet 23cba5cc51d7414a9352caedcc362c9c
0115 14:52:04.636377 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.670216 (+ 33839us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.682020 (+ 11804us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.703622 (+ 21602us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8620,"tablet-delete.queue_time_us":1723937}
I20250115 14:52:04.704559 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_38d8cb2a1e1644e989325bc288b09b5d" } new_table_name: "default.Table_5d9f72946dd846b98daa1df0c392cb72" modify_external_catalogs: true
I20250115 14:52:04.704634 1167 tablet_replica.cc:331] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
W20250115 14:52:04.704771 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a69fe39cbd8f4a628213badbaafff14a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.704808 1167 raft_consensus.cc:2238] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:04.705063 1167 pending_rounds.cc:62] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:04.705106 1167 pending_rounds.cc:69] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558354792718336 op_type: NO_OP noop_request { }
I20250115 14:52:04.705191 1167 raft_consensus.cc:2883] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:04.705266 1167 raft_consensus.cc:2267] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.705788 1167 ts_tablet_manager.cc:1905] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.706297 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ca5cc55b50ee4953800f10efe0ffb546
I20250115 14:52:04.706777 448 catalog_manager.cc:6596] ReplaceTablet: tablet ca5cc55b50ee4953800f10efe0ffb546 deleted and replaced by tablet d443d39716d04b45b3151be80ed4320b
I20250115 14:52:04.707015 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 23cba5cc51d7414a9352caedcc362c9c (table default.Table_27cd4d793c8f4e77aa531c06dfa74a2d [id=a6994f260e1a4eada78d8c9e9fcf334e]) successfully deleted
W20250115 14:52:04.707165 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33193ebbb6e34f1785a1687f78ff1258 has no consensus state
I20250115 14:52:04.709252 751 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5cc55b50ee4953800f10efe0ffb546 with delete_type TABLET_DATA_DELETED (replaced by tablet d443d39716d04b45b3151be80ed4320b) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.707333 1168 tablet_bootstrap.cc:654] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.707603 893 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5cc55b50ee4953800f10efe0ffb546 with delete_type TABLET_DATA_DELETED (replaced by tablet d443d39716d04b45b3151be80ed4320b) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.711783 1843 consensus_queue.cc:1035] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:52:04.713909 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 929830d8e8fd4b0c8a7b18c941706929 consensus state has no leader
I20250115 14:52:04.716773 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b15de9bf8943cdb9e5abd943484993 with delete_type TABLET_DATA_DELETED (replaced by tablet 1de35551e26643b3b1e570ab8b2f6f56) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:04.718268 848 consensus_peers.cc:487] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: caed629878594954842dfc6a40cc8b6d. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.723269 1166 ts_tablet_manager.cc:1918] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:04.723672 1166 log.cc:1198] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/a915e0c0a30940ed8a6233f117e2a560
I20250115 14:52:04.725260 1166 ts_tablet_manager.cc:1939] T a915e0c0a30940ed8a6233f117e2a560 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:04.726411 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 010e6535b6bc44f391e6f2b6f1686532 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d891d568574c51b0d062ca9ac77f45) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.727067 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]) successfully deleted
W20250115 14:52:04.730422 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 105b6148ae944e5c906e07d987177cc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 105b6148ae944e5c906e07d987177cc5 already in progress: creating tablet
I20250115 14:52:04.731596 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 50273a0019bb4cd8977f7ef014372d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 70425c25919b4d0b999450210e1cb33f) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.731818 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ca5cc55b50ee4953800f10efe0ffb546 with delete_type TABLET_DATA_DELETED (replaced by tablet d443d39716d04b45b3151be80ed4320b) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:04.731842 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 74d00bcee22d40769102eacaf728480d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74d00bcee22d40769102eacaf728480d already in progress: creating tablet
I20250115 14:52:04.732070 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b15de9bf8943cdb9e5abd943484993 with delete_type TABLET_DATA_DELETED (replaced by tablet 1de35551e26643b3b1e570ab8b2f6f56) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:04.732854 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9494c0c57be8499a883888d067c7db77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9494c0c57be8499a883888d067c7db77 already in progress: creating tablet
I20250115 14:52:04.733500 2117 raft_consensus.cc:491] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.733632 2117 raft_consensus.cc:513] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:04.733914 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 282d988bc41443c381bb0ba69b618f5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 282d988bc41443c381bb0ba69b618f5d already in progress: creating tablet
I20250115 14:52:04.734259 2117 leader_election.cc:290] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.734696 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171643c40c9b43ffb823f7c33789284d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:04.734901 911 raft_consensus.cc:2463] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:04.735458 710 leader_election.cc:304] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:04.735817 2117 raft_consensus.cc:2798] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:04.735886 2117 raft_consensus.cc:491] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:04.735941 2117 raft_consensus.cc:3054] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.735865 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 01291d41a2b345b4bfd00e6be0fb0129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.739069 2117 raft_consensus.cc:513] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.740025 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171643c40c9b43ffb823f7c33789284d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:04.740891 1047 raft_consensus.cc:2463] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:04.742630 1751 ts_tablet_manager.cc:1918] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.742730 1751 log.cc:1198] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9d8c53fdf16243f0a614e48ec4ac94f4
I20250115 14:52:04.743345 1751 ts_tablet_manager.cc:1939] T 9d8c53fdf16243f0a614e48ec4ac94f4 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:04.743853 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9d8c53fdf16243f0a614e48ec4ac94f4 (table default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]) successfully deleted
I20250115 14:52:04.743849 2117 leader_election.cc:290] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.744191 1751 tablet_replica.cc:331] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:04.744410 1751 raft_consensus.cc:2238] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.744573 1751 raft_consensus.cc:2267] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.745088 1751 ts_tablet_manager.cc:1905] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.745074 1167 ts_tablet_manager.cc:1918] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:04.745178 1167 log.cc:1198] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e3fbde011216426cb41c51fe359dcb5f
I20250115 14:52:04.746122 1167 ts_tablet_manager.cc:1939] T e3fbde011216426cb41c51fe359dcb5f P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:04.746575 1166 tablet_replica.cc:331] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:04.747507 1162 tablet_bootstrap.cc:492] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:04.747731 1162 ts_tablet_manager.cc:1397] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.350s user 0.032s sys 0.051s
I20250115 14:52:04.747618 1166 raft_consensus.cc:2238] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:04.748107 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 398ac8af619c427287b3964443173864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 398ac8af619c427287b3964443173864 already in progress: creating tablet
W20250115 14:52:04.748431 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 48b15de9bf8943cdb9e5abd943484993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48b15de9bf8943cdb9e5abd943484993 already in progress: creating tablet
I20250115 14:52:04.748353 1166 raft_consensus.cc:2267] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.749046 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171643c40c9b43ffb823f7c33789284d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:04.749207 1047 raft_consensus.cc:3054] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.751504 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "171643c40c9b43ffb823f7c33789284d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:04.751677 911 raft_consensus.cc:3054] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:04.753736 1166 ts_tablet_manager.cc:1905] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.754207 911 raft_consensus.cc:2463] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:04.755316 710 leader_election.cc:304] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:04.755599 2117 raft_consensus.cc:2798] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:04.755702 2117 raft_consensus.cc:695] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:04.755928 2117 consensus_queue.cc:237] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.757858 1047 raft_consensus.cc:2463] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:04.762249 1162 raft_consensus.cc:357] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.762601 1162 raft_consensus.cc:383] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.762725 1162 raft_consensus.cc:738] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.770601 1164 tablet_bootstrap.cc:492] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:04.772692 2079 tablet.cc:1721] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:04.773011 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c6d900c973594576916cf9d2b3a730b0" } modify_external_catalogs: true
I20250115 14:52:04.774417 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_28021473fc1344df97d1fdd68b2023dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.775024 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28021473fc1344df97d1fdd68b2023dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.775321 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_458a573b4a4b4714950723b426f0d265" } modify_external_catalogs: true
I20250115 14:52:04.777858 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d3d1424de57f42b89ddb3d6f1cbff283" } modify_external_catalogs: true
I20250115 14:52:04.778218 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6da53a816dfb4f40a1849c11399d8341"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.778728 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6da53a816dfb4f40a1849c11399d8341 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.769820 1162 consensus_queue.cc:260] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.782462 1162 ts_tablet_manager.cc:1428] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20250115 14:52:04.782714 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:04.782950 1162 tablet_bootstrap.cc:492] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:04.785523 1162 tablet_bootstrap.cc:654] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.775023 1164 ts_tablet_manager.cc:1397] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.244s user 0.000s sys 0.004s
I20250115 14:52:04.787128 1164 raft_consensus.cc:357] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.787487 1164 raft_consensus.cc:383] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.787613 1164 raft_consensus.cc:738] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.787954 1164 consensus_queue.cc:260] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.788957 1164 ts_tablet_manager.cc:1428] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:04.749151 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 601) took 1793 ms. Trace:
I20250115 14:52:04.805441 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7]). Expected version 2 got 0
I20250115 14:52:04.805527 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c6d900c973594576916cf9d2b3a730b0" } modify_external_catalogs: true
I20250115 14:52:04.807561 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d3d1424de57f42b89ddb3d6f1cbff283" } modify_external_catalogs: true
I20250115 14:52:04.807960 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet dff155a5c2b0479a90e74993de6f402d (table default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]). Expected version 1 got 0
I20250115 14:52:04.808148 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:04.808362 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7]). Expected version 2 got 0
I20250115 14:52:04.808305 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ff85ca99bc044a8c8df53fe654e0cf14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.808846 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff85ca99bc044a8c8df53fe654e0cf14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.809450 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_201c8e612d7d4a85a6d4d4b0a5095254"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.809806 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_458a573b4a4b4714950723b426f0d265" } modify_external_catalogs: true
W20250115 14:52:04.809952 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_201c8e612d7d4a85a6d4d4b0a5095254 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.811136 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_57e02beccdd64ec6ad08a1f3d0908b65" } modify_external_catalogs: true
I20250115 14:52:04.811272 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_57e02beccdd64ec6ad08a1f3d0908b65" } modify_external_catalogs: true
W20250115 14:52:04.811672 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6f3f8cae6244968927b864e7ce97565 consensus state has no leader
I20250115 14:52:04.816159 1166 ts_tablet_manager.cc:1918] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.820639 879 tablet_service.cc:1514] Processing DeleteTablet for tablet caa1827366a34f039fd12deb855f07b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:04.823614 848 consensus_peers.cc:487] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a915e0c0a30940ed8a6233f117e2a560. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.826946 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:04.829902 2079 tablet.cc:1721] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:04.831251 2131 tablet.cc:1721] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:04.834003 2137 raft_consensus.cc:2743] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.800259 1167 rpcz_store.cc:276] 0115 14:52:02.955992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:02.956050 (+ 58us) service_pool.cc:225] Handling call
0115 14:52:04.704579 (+1748529us) ts_tablet_manager.cc:1145] Deleting tablet e3fbde011216426cb41c51fe359dcb5f
0115 14:52:04.704588 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.720750 (+ 16162us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.744970 (+ 24220us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.749133 (+ 4163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1748358}
I20250115 14:52:04.836395 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet e3fbde011216426cb41c51fe359dcb5f (table default.Table_1cd00e265b464348bd1df215a1a0d9e9 [id=3fe917c536db4535b79e95a59f9ea8f8]) successfully deleted
I20250115 14:52:04.836661 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 605) took 1815 ms. Trace:
I20250115 14:52:04.836867 1751 ts_tablet_manager.cc:1918] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:04.836968 1751 log.cc:1198] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9853dcdfee5f43de84798cfa0bef674e
I20250115 14:52:04.836858 1167 rpcz_store.cc:276] 0115 14:52:03.021478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.021549 (+ 71us) service_pool.cc:225] Handling call
0115 14:52:04.836460 (+1814911us) ts_tablet_manager.cc:1145] Deleting tablet e5a059e2f0654a4aafe9e8082b8fde7a
0115 14:52:04.836497 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.836645 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1814720}
I20250115 14:52:04.837229 1751 ts_tablet_manager.cc:1939] T 9853dcdfee5f43de84798cfa0bef674e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:04.837464 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e5a059e2f0654a4aafe9e8082b8fde7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5a059e2f0654a4aafe9e8082b8fde7a already in progress: creating tablet
I20250115 14:52:04.816262 1166 log.cc:1198] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d195be2d543f4a8693045a024912c43a
I20250115 14:52:04.837669 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 606) took 1814 ms. Trace:
I20250115 14:52:04.837929 1166 ts_tablet_manager.cc:1939] T d195be2d543f4a8693045a024912c43a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:04.837831 1167 rpcz_store.cc:276] 0115 14:52:03.022736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.022789 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:04.837486 (+1814697us) ts_tablet_manager.cc:1145] Deleting tablet ba596cfb3ce64718a42a9083975edc8d
0115 14:52:04.837520 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.837627 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1814554}
W20250115 14:52:04.838336 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ba596cfb3ce64718a42a9083975edc8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba596cfb3ce64718a42a9083975edc8d already in progress: creating tablet
I20250115 14:52:04.838419 1167 tablet_replica.cc:331] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:04.838692 1167 raft_consensus.cc:2238] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:04.838825 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d195be2d543f4a8693045a024912c43a (table default.Table_bba588d115d34226951843abdb9b6ebe [id=e99d33d2241343e2818d4217babf9a49]) successfully deleted
I20250115 14:52:04.839228 1167 pending_rounds.cc:62] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:04.839349 1167 pending_rounds.cc:69] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558356353708032 op_type: NO_OP noop_request { }
W20250115 14:52:04.839468 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 496dec75500347699625a755dceed2bb because the tablet was not found. No further retry: Not found: Tablet not found: 496dec75500347699625a755dceed2bb
I20250115 14:52:04.839568 1167 raft_consensus.cc:2883] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:04.839728 1167 raft_consensus.cc:2267] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:04.840011 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ca5cc55b50ee4953800f10efe0ffb546 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca5cc55b50ee4953800f10efe0ffb546 already in progress: creating tablet
W20250115 14:52:04.834079 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 180879us
I20250115 14:52:04.842500 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9853dcdfee5f43de84798cfa0bef674e (table default.Table_4ec686b46a094818bd820c4f3854220b [id=5dc3e53ee73a4441960dc231d02d163f]) successfully deleted
W20250115 14:52:04.843124 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa already in progress: creating tablet
W20250115 14:52:04.843467 847 consensus_peers.cc:487] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: caed629878594954842dfc6a40cc8b6d. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:04.843729 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 342bf8d4dc4244f982b1b63aaae0fb45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 342bf8d4dc4244f982b1b63aaae0fb45 already in progress: creating tablet
W20250115 14:52:04.844244 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 655489ad9bd34be0a8283a295f583092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 655489ad9bd34be0a8283a295f583092 already in progress: creating tablet
I20250115 14:52:04.844427 1164 tablet_bootstrap.cc:492] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:04.845405 1751 tablet_replica.cc:331] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:04.846197 1751 raft_consensus.cc:2238] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:04.847962 1164 tablet_bootstrap.cc:654] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.848330 1751 raft_consensus.cc:2267] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.851281 1168 tablet_bootstrap.cc:492] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:04.851512 1751 ts_tablet_manager.cc:1905] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.853677 1168 ts_tablet_manager.cc:1397] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.158s user 0.008s sys 0.014s
I20250115 14:52:04.854374 1168 raft_consensus.cc:357] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.854633 1168 raft_consensus.cc:383] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.854720 1168 raft_consensus.cc:738] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.854970 1168 consensus_queue.cc:260] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.855656 1168 ts_tablet_manager.cc:1428] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:04.855930 1168 tablet_bootstrap.cc:492] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:04.860359 1751 ts_tablet_manager.cc:1918] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:04.860718 1751 log.cc:1198] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4821224ac8f1472d8b1e8afc06a118a3
I20250115 14:52:04.861088 1751 ts_tablet_manager.cc:1939] T 4821224ac8f1472d8b1e8afc06a118a3 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:04.861207 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 83415f1c07ad495cabd287bdcd54dad4
I20250115 14:52:04.862746 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 83415f1c07ad495cabd287bdcd54dad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.862947 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 83415f1c07ad495cabd287bdcd54dad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.862985 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a48a0af9422f43cf86531665c0fa1aee
I20250115 14:52:04.863293 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 83415f1c07ad495cabd287bdcd54dad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.863869 448 master_service.cc:946] ReplaceTablet: received request to replace tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:04.864215 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet a48a0af9422f43cf86531665c0fa1aee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.864904 879 tablet_service.cc:1514] Processing DeleteTablet for tablet a48a0af9422f43cf86531665c0fa1aee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:04.865518 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 83415f1c07ad495cabd287bdcd54dad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83415f1c07ad495cabd287bdcd54dad4 already in progress: creating tablet
I20250115 14:52:04.866652 741 tablet_service.cc:1514] Processing DeleteTablet for tablet a48a0af9422f43cf86531665c0fa1aee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:04.867511 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a48a0af9422f43cf86531665c0fa1aee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48a0af9422f43cf86531665c0fa1aee already in progress: creating tablet
I20250115 14:52:04.868453 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7]). Expected version 2 got 0
I20250115 14:52:04.869518 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet cc9ce226d3ba44d6908257bbd0df3a12
I20250115 14:52:04.870904 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9ce226d3ba44d6908257bbd0df3a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.871470 879 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9ce226d3ba44d6908257bbd0df3a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.871698 741 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9ce226d3ba44d6908257bbd0df3a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.872278 2149 tablet.cc:1721] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:04.872680 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c37f4c2ab9d248a1af817cfa2c2e2ff0
I20250115 14:52:04.873131 448 catalog_manager.cc:6596] ReplaceTablet: tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 deleted and replaced by tablet bc1440e3cd444cf8b3365a0af8de8958
I20250115 14:52:04.873534 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7]). Expected version 2 got 0
I20250115 14:52:04.874141 1162 tablet_bootstrap.cc:492] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:04.874257 1162 ts_tablet_manager.cc:1397] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.002s
I20250115 14:52:04.874799 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 with delete_type TABLET_DATA_DELETED (replaced by tablet bc1440e3cd444cf8b3365a0af8de8958) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.875442 879 tablet_service.cc:1514] Processing DeleteTablet for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 with delete_type TABLET_DATA_DELETED (replaced by tablet bc1440e3cd444cf8b3365a0af8de8958) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:04.876318 1167 ts_tablet_manager.cc:1905] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.878311 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet dff155a5c2b0479a90e74993de6f402d (table default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]). Expected version 1 got 0
I20250115 14:52:04.878334 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4c53ae523be9439e825de4b84817ff71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.879588 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c53ae523be9439e825de4b84817ff71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.881382 741 tablet_service.cc:1514] Processing DeleteTablet for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 with delete_type TABLET_DATA_DELETED (replaced by tablet bc1440e3cd444cf8b3365a0af8de8958) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.882120 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ce65604c294021af93632ed073f627 with delete_type TABLET_DATA_DELETED (replaced by tablet 339dd408dab7430c980058af3af04433) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.883057 2137 raft_consensus.cc:491] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.883177 2137 raft_consensus.cc:513] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.883411 1162 raft_consensus.cc:357] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.883576 1162 raft_consensus.cc:383] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.883622 1162 raft_consensus.cc:738] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.883739 2137 leader_election.cc:290] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.883793 1162 consensus_queue.cc:260] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.884559 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8ab6a34caa486ab929af2cac3ce957" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:04.884775 2149 tablet.cc:1721] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:52:04.885026 983 leader_election.cc:343] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.884258 1162 ts_tablet_manager.cc:1428] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250115 14:52:04.885396 1162 tablet_bootstrap.cc:492] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:04.885673 984 leader_election.cc:343] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.886113 984 leader_election.cc:304] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.886647 2137 raft_consensus.cc:2743] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.886759 1162 tablet_bootstrap.cc:654] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.888463 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8ab6a34caa486ab929af2cac3ce957" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:04.890758 984 leader_election.cc:343] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.891067 984 leader_election.cc:304] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.891479 2137 raft_consensus.cc:2743] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.891840 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:04.898108 2137 raft_consensus.cc:491] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.898213 2137 raft_consensus.cc:513] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.898706 2137 leader_election.cc:290] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.899277 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ce65604c294021af93632ed073f627" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:04.899801 983 leader_election.cc:343] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.897965 1168 tablet_bootstrap.cc:654] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:04.901314 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet cc9ce226d3ba44d6908257bbd0df3a12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9ce226d3ba44d6908257bbd0df3a12 already in progress: creating tablet
W20250115 14:52:04.901616 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 already in progress: creating tablet
I20250115 14:52:04.901830 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6da53a816dfb4f40a1849c11399d8341" } modify_external_catalogs: true
I20250115 14:52:04.902042 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6da53a816dfb4f40a1849c11399d8341" } modify_external_catalogs: true
I20250115 14:52:04.903645 455 master_service.cc:946] ReplaceTablet: received request to replace tablet fa57c1fa96594a40b2551ad503127e4f from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:04.904958 1164 tablet_bootstrap.cc:492] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:04.905086 1164 ts_tablet_manager.cc:1397] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20250115 14:52:04.906870 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9daeb4cb953b43a6bb9dc34b756d7716" } new_table_name: "default.Table_a5c4980506e643c7918b795c31ebfbb0" modify_external_catalogs: true
I20250115 14:52:04.908411 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_41d5e114b3da4fca97f9efc5a69dfe2f" } new_table_name: "default.Table_8f606b5f353c41be81616521eb8423f3" modify_external_catalogs: true
I20250115 14:52:04.909235 1162 tablet_bootstrap.cc:492] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:04.909358 1162 ts_tablet_manager.cc:1397] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
W20250115 14:52:04.909441 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2e4c2707915e4e97ad6c66cb249c6d9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4c2707915e4e97ad6c66cb249c6d9e already in progress: creating tablet
W20250115 14:52:04.909993 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e180e37a1d5d46d78380323662b80f85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e180e37a1d5d46d78380323662b80f85 already in progress: creating tablet
I20250115 14:52:04.910285 1162 raft_consensus.cc:357] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.910456 1162 raft_consensus.cc:383] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.910514 1162 raft_consensus.cc:738] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.910696 1162 consensus_queue.cc:260] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:04.911085 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c1a77d07b1c946b18e0769a37d8a50a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1a77d07b1c946b18e0769a37d8a50a1 already in progress: creating tablet
I20250115 14:52:04.911237 1162 ts_tablet_manager.cc:1428] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:04.911463 1162 tablet_bootstrap.cc:492] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:04.911397 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ce65604c294021af93632ed073f627" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:04.912941 1168 tablet_bootstrap.cc:492] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:04.913057 1168 ts_tablet_manager.cc:1397] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.000s
I20250115 14:52:04.913640 1168 raft_consensus.cc:357] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.913815 1168 raft_consensus.cc:383] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.913856 1168 raft_consensus.cc:738] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.914017 1168 consensus_queue.cc:260] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:04.914448 1168 ts_tablet_manager.cc:1428] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:04.914681 1168 tablet_bootstrap.cc:492] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:04.915913 1168 tablet_bootstrap.cc:654] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.915865 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c9a1822ec7ae455ba4b2d5a1181d3eb1" } new_table_name: "default.Table_924719cc6f6a45fc99e425268591354c" modify_external_catalogs: true
I20250115 14:52:04.917310 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_469367eced4344d2bbb224143a1d6e67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:04.917785 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_469367eced4344d2bbb224143a1d6e67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.920104 1162 tablet_bootstrap.cc:654] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:04.922389 984 leader_election.cc:343] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.922971 984 leader_election.cc:304] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.923238 2137 raft_consensus.cc:2743] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.925407 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet fa57c1fa96594a40b2551ad503127e4f
I20250115 14:52:04.925904 455 catalog_manager.cc:6596] ReplaceTablet: tablet fa57c1fa96594a40b2551ad503127e4f deleted and replaced by tablet 9c35882d2e5c464c9cfad496ba9d6af9
I20250115 14:52:04.926270 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1e83e61b373d4455b007c7214a0dba53"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:04.926735 879 tablet_service.cc:1514] Processing DeleteTablet for tablet fa57c1fa96594a40b2551ad503127e4f with delete_type TABLET_DATA_DELETED (replaced by tablet 9c35882d2e5c464c9cfad496ba9d6af9) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:04.926784 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e83e61b373d4455b007c7214a0dba53 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:04.926945 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet fa57c1fa96594a40b2551ad503127e4f with delete_type TABLET_DATA_DELETED (replaced by tablet 9c35882d2e5c464c9cfad496ba9d6af9) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.927820 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 1e5648b2042e46f8b009b6ad28a64aee
I20250115 14:52:04.928031 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed] alter complete (version 1)
I20250115 14:52:04.928148 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9b06f855e23e47cbb239075f101b585c" } new_table_name: "default.Table_adcff7acc18b46e79d647e28af423fbd" modify_external_catalogs: true
I20250115 14:52:04.928680 741 tablet_service.cc:1514] Processing DeleteTablet for tablet fa57c1fa96594a40b2551ad503127e4f with delete_type TABLET_DATA_DELETED (replaced by tablet 9c35882d2e5c464c9cfad496ba9d6af9) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:04.929615 2137 raft_consensus.cc:491] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.929730 2137 raft_consensus.cc:513] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.930290 2137 leader_election.cc:290] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.930816 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b75d759f834d558c94cc981e338ac0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:04.931313 983 leader_election.cc:343] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:04.931370 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 338706b2d7514f94afd3579da458b4ec consensus state has no leader
W20250115 14:52:04.931746 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d03eb225fad7459a955a3ec5e0002be1 consensus state has no leader
W20250115 14:52:04.932376 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c consensus state has no leader
I20250115 14:52:04.932708 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_65de108282934cad838f2c1f716989a5 [id=be99ec07535243a68c1b3ee1be0c62e7] alter complete (version 2)
W20250115 14:52:04.932377 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet fa57c1fa96594a40b2551ad503127e4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa57c1fa96594a40b2551ad503127e4f already in progress: creating tablet
I20250115 14:52:04.933506 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 759a4ebe7b0444128ddd31a78aad463e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.940021 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abde784f9c948cb96e6cc93c30817bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:04.940619 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b75d759f834d558c94cc981e338ac0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:04.941262 984 leader_election.cc:343] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.941372 984 leader_election.cc:304] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.942306 2137 raft_consensus.cc:2743] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.947966 1167 ts_tablet_manager.cc:1918] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:04.948885 1167 log.cc:1198] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b2c385edf4414157845b971aaf366831
W20250115 14:52:04.950026 847 consensus_peers.cc:487] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc8337d181d44ac7bb82551a677b6831. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.951828 1167 ts_tablet_manager.cc:1939] T b2c385edf4414157845b971aaf366831 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:04.954095 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 607) took 1911 ms. Trace:
W20250115 14:52:04.954623 847 consensus_peers.cc:487] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9e5bc6131944fcab9c550caac4f8d85. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.954981 1167 rpcz_store.cc:276] 0115 14:52:03.042369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.042472 (+ 103us) service_pool.cc:225] Handling call
0115 14:52:04.838354 (+1795882us) ts_tablet_manager.cc:1145] Deleting tablet b2c385edf4414157845b971aaf366831
0115 14:52:04.838363 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.899797 (+ 61434us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.947346 (+ 47549us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:04.953853 (+ 6507us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1795723}
I20250115 14:52:04.956518 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b2c385edf4414157845b971aaf366831 (table default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]) successfully deleted
W20250115 14:52:04.957780 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 11fddbd9ba14443a91865387e1d1c238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11fddbd9ba14443a91865387e1d1c238 already in progress: creating tablet
I20250115 14:52:04.957772 1167 tablet_replica.cc:331] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
W20250115 14:52:04.958379 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ed777d7496514ec5986a7e6f53d532b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed777d7496514ec5986a7e6f53d532b1 already in progress: creating tablet
I20250115 14:52:04.958626 1751 tablet_replica.cc:331] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:04.958796 1751 raft_consensus.cc:2238] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:04.958947 1751 raft_consensus.cc:2267] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.959448 1751 ts_tablet_manager.cc:1905] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:04.959594 1167 raft_consensus.cc:2238] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
W20250115 14:52:04.962172 848 consensus_peers.cc:487] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc8337d181d44ac7bb82551a677b6831. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:04.961951 1167 raft_consensus.cc:2267] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:04.962467 1164 raft_consensus.cc:357] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.963092 1164 raft_consensus.cc:383] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.963169 1164 raft_consensus.cc:738] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.963393 1164 consensus_queue.cc:260] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.965081 1164 ts_tablet_manager.cc:1428] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20250115 14:52:04.965373 1164 tablet_bootstrap.cc:492] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:04.966583 1167 ts_tablet_manager.cc:1905] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:04.967691 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 919ab6ec6f3745e082aa441dfebc235e consensus state has no leader
I20250115 14:52:04.968009 2137 raft_consensus.cc:491] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:04.968125 2137 raft_consensus.cc:513] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.968691 2137 leader_election.cc:290] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:04.969108 2083 consensus_queue.cc:1035] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:04.969247 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abde784f9c948cb96e6cc93c30817bd" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:04.969791 983 leader_election.cc:343] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.971222 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abde784f9c948cb96e6cc93c30817bd" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:04.971993 984 leader_election.cc:343] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:04.972095 984 leader_election.cc:304] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:04.972385 1876 raft_consensus.cc:2743] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:04.973399 1164 tablet_bootstrap.cc:654] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:04.981878 1162 tablet_bootstrap.cc:492] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:04.982033 1162 ts_tablet_manager.cc:1397] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.001s
I20250115 14:52:04.989521 1162 raft_consensus.cc:357] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.989701 1162 raft_consensus.cc:383] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.989744 1162 raft_consensus.cc:738] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.989924 1162 consensus_queue.cc:260] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.990379 1162 ts_tablet_manager.cc:1428] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.008s user 0.000s sys 0.000s
I20250115 14:52:04.990599 1162 tablet_bootstrap.cc:492] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:04.992731 1164 tablet_bootstrap.cc:492] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:04.992874 1164 ts_tablet_manager.cc:1397] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250115 14:52:04.993582 1164 raft_consensus.cc:357] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.993773 1164 raft_consensus.cc:383] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:04.993817 1164 raft_consensus.cc:738] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:04.996847 1164 consensus_queue.cc:260] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:04.998944 1164 ts_tablet_manager.cc:1428] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250115 14:52:04.999215 1164 tablet_bootstrap.cc:492] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.000787 1164 tablet_bootstrap.cc:654] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.002887 1167 ts_tablet_manager.cc:1918] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:05.003052 1167 log.cc:1198] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/a3a7a487a42949bd9641c38d6c1fcfd3
I20250115 14:52:05.003516 1167 ts_tablet_manager.cc:1939] T a3a7a487a42949bd9641c38d6c1fcfd3 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:05.003865 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 608) took 1898 ms. Trace:
I20250115 14:52:05.003985 1167 rpcz_store.cc:276] 0115 14:52:03.104914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.104964 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:04.957278 (+1852314us) ts_tablet_manager.cc:1145] Deleting tablet a3a7a487a42949bd9641c38d6c1fcfd3
0115 14:52:04.957513 (+ 235us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:04.990397 (+ 32884us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.002792 (+ 12395us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.003851 (+ 1059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1852144}
I20250115 14:52:05.004518 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet a3a7a487a42949bd9641c38d6c1fcfd3 (table default.Table_8b9b25dd8a85434a83d2641ffbc5189e [id=738170c5571a4d958944ed432428e99d]) successfully deleted
I20250115 14:52:05.004889 1167 tablet_replica.cc:331] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:05.005071 1167 raft_consensus.cc:2238] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:05.005396 1167 pending_rounds.cc:62] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:05.005479 1167 pending_rounds.cc:69] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558356668788736 op_type: NO_OP noop_request { }
I20250115 14:52:05.005621 1167 raft_consensus.cc:2883] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:05.005740 1167 raft_consensus.cc:2267] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.006263 1167 ts_tablet_manager.cc:1905] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.010958 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fc7747535f7b4449867ef0aeb96f88f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:05.011241 452 master_service.cc:946] ReplaceTablet: received request to replace tablet fb81e8fcc38a41e69cfb0e34dbaaac84 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.011418 1168 tablet_bootstrap.cc:492] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.011583 1168 ts_tablet_manager.cc:1397] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.003s
I20250115 14:52:05.011775 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c4f5e92e6f994cb0ad035e3cef60b3ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:05.012269 1168 raft_consensus.cc:357] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.012447 1168 raft_consensus.cc:383] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.012488 1168 raft_consensus.cc:738] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.013226 1168 consensus_queue.cc:260] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.013772 1168 ts_tablet_manager.cc:1428] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250115 14:52:05.014135 1751 ts_tablet_manager.cc:1918] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:05.014274 1751 log.cc:1198] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a915e0c0a30940ed8a6233f117e2a560
I20250115 14:52:05.014789 1751 ts_tablet_manager.cc:1939] T a915e0c0a30940ed8a6233f117e2a560 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:05.016158 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4f5e92e6f994cb0ad035e3cef60b3ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.016623 1168 tablet_bootstrap.cc:492] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:05.017917 1168 tablet_bootstrap.cc:654] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:05.019769 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc7747535f7b4449867ef0aeb96f88f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.021674 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_28d063764bb94657a5f118c8820e8fbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.022153 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28d063764bb94657a5f118c8820e8fbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.023757 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0763d4939e83417ab72a4b2fc39d2f33" } new_table_name: "default.Table_61edbe7db3b44d02a60d60d865a86ec5" modify_external_catalogs: true
I20250115 14:52:05.029987 1164 tablet_bootstrap.cc:492] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:05.030133 1164 ts_tablet_manager.cc:1397] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
W20250115 14:52:05.030737 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 288b0e455a854464940f65a2ab909d0b has no consensus state
I20250115 14:52:05.030943 1164 raft_consensus.cc:357] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.034288 1167 ts_tablet_manager.cc:1918] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:05.035250 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]) successfully deleted
I20250115 14:52:05.036175 1167 log.cc:1198] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/bc8337d181d44ac7bb82551a677b6831
I20250115 14:52:05.037786 1162 tablet_bootstrap.cc:654] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.038316 1167 ts_tablet_manager.cc:1939] T bc8337d181d44ac7bb82551a677b6831 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:05.039273 1751 tablet_replica.cc:331] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:05.039491 1751 raft_consensus.cc:2238] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.039675 1751 raft_consensus.cc:2267] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.040186 1751 ts_tablet_manager.cc:1905] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:05.035552 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 282d988bc41443c381bb0ba69b618f5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 282d988bc41443c381bb0ba69b618f5d already in progress: creating tablet
W20250115 14:52:05.042240 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 105b6148ae944e5c906e07d987177cc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 105b6148ae944e5c906e07d987177cc5 already in progress: creating tablet
I20250115 14:52:05.043570 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 609) took 1928 ms. Trace:
I20250115 14:52:05.031132 1164 raft_consensus.cc:383] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.044597 1164 raft_consensus.cc:738] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.044617 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
W20250115 14:52:05.042929 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 74d00bcee22d40769102eacaf728480d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74d00bcee22d40769102eacaf728480d already in progress: creating tablet
I20250115 14:52:05.045444 1168 tablet_bootstrap.cc:492] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.045555 1168 ts_tablet_manager.cc:1397] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.002s
W20250115 14:52:05.045545 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 398ac8af619c427287b3964443173864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 398ac8af619c427287b3964443173864 already in progress: creating tablet
W20250115 14:52:05.045807 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9494c0c57be8499a883888d067c7db77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9494c0c57be8499a883888d067c7db77 already in progress: creating tablet
I20250115 14:52:05.046228 1168 raft_consensus.cc:357] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.046443 1168 raft_consensus.cc:383] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.046489 1168 raft_consensus.cc:738] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.046669 1168 consensus_queue.cc:260] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.045678 446 catalog_manager.cc:5526] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d7ecece2ab49e1be7079216517d13b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d2d7ecece2ab49e1be7079216517d13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:05.047092 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_a08bf8546b9148dc97c582984b4409c8 [id=7628dbb6818f443680a97fef665d6d2b]). Expected version 3 got 0
I20250115 14:52:05.047209 1168 ts_tablet_manager.cc:1428] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:05.047287 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]). Expected version 1 got 0
I20250115 14:52:05.047467 1168 tablet_bootstrap.cc:492] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:05.047549 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:05.049058 451 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 009ddfe9dfe14ba2b067f907a9d13538 (table default.Table_96eab6cd20b842138b5acc7c8d0ac5d6 [id=816d14738e72406a88b9eed42d72be73]). Expected version 2 got 0
I20250115 14:52:05.049260 451 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 338c0f6f39fd4335bbcda733b28b8bf8 (table default.Table_f367ac1a82704324a73f7079165a6e45 [id=6756cde43edd4d1e8a2768c101c4aa19]). Expected version 3 got 0
I20250115 14:52:05.049678 1164 consensus_queue.cc:260] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.051234 1164 ts_tablet_manager.cc:1428] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.021s user 0.002s sys 0.001s
I20250115 14:52:05.051498 1164 tablet_bootstrap.cc:492] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.052629 1168 tablet_bootstrap.cc:654] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.043948 1167 rpcz_store.cc:276] 0115 14:52:03.114700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.114743 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:05.004839 (+1890096us) ts_tablet_manager.cc:1145] Deleting tablet bc8337d181d44ac7bb82551a677b6831
0115 14:52:05.004850 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.010285 (+ 5435us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.034184 (+ 23899us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.043394 (+ 9210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1889945}
I20250115 14:52:05.054167 1164 tablet_bootstrap.cc:654] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.054615 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 610) took 1880 ms. Trace:
I20250115 14:52:05.056385 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet bc8337d181d44ac7bb82551a677b6831 (table default.Table_9502b06eea13418c8ce173b8f4b5ec08 [id=cca81f1c67d248aebdd936fff8831e88]) successfully deleted
I20250115 14:52:05.057045 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_00fd62cc66984eecb416652888d31733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.057536 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00fd62cc66984eecb416652888d31733 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.056797 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc975b044dd439dbd1f8f9a82cce751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.057785 1167 rpcz_store.cc:276] 0115 14:52:03.173668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.173718 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:05.054507 (+1880789us) ts_tablet_manager.cc:1145] Deleting tablet 4934bdde6023419fafd750920bcfe3d4
0115 14:52:05.054516 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.054599 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880631}
I20250115 14:52:05.058337 1751 ts_tablet_manager.cc:1918] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.058429 1167 tablet_replica.cc:331] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:05.059190 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0a2bcfc5c5714a3eac85b6a8beafbf59
I20250115 14:52:05.059394 741 tablet_service.cc:1467] Processing CreateTablet for tablet 79591d616d40427b9a6f321fed5533ac (DEFAULT_TABLE table=default.Table_3f9ece922e604339970891650343338a [id=0a5921c4f0814b20b86e3d9fa45f9c82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.059744 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79591d616d40427b9a6f321fed5533ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.060027 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b21a9aaeef5245b2813b04521d7a113b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.060483 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b21a9aaeef5245b2813b04521d7a113b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.059682 879 tablet_service.cc:1467] Processing CreateTablet for tablet 79591d616d40427b9a6f321fed5533ac (DEFAULT_TABLE table=default.Table_3f9ece922e604339970891650343338a [id=0a5921c4f0814b20b86e3d9fa45f9c82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.060895 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.061128 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79591d616d40427b9a6f321fed5533ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.060581 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.060231 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.061676 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 79591d616d40427b9a6f321fed5533ac (DEFAULT_TABLE table=default.Table_3f9ece922e604339970891650343338a [id=0a5921c4f0814b20b86e3d9fa45f9c82]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.062156 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79591d616d40427b9a6f321fed5533ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.063580 893 tablet_service.cc:1467] Processing CreateTablet for tablet 23ad6f98f8fa445ab2100f29a7ffb991 (DEFAULT_TABLE table=default.Table_29e435c06be34d20b3fe6ab2745835d8 [id=7caed57491e248ee8f5ac239666c8a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.063882 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ad6f98f8fa445ab2100f29a7ffb991. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.064648 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 already in progress: creating tablet
I20250115 14:52:05.065614 755 tablet_service.cc:1467] Processing CreateTablet for tablet 23ad6f98f8fa445ab2100f29a7ffb991 (DEFAULT_TABLE table=default.Table_29e435c06be34d20b3fe6ab2745835d8 [id=7caed57491e248ee8f5ac239666c8a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.065830 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_28a52958055546da964198b7c17f8b8c" } modify_external_catalogs: true
I20250115 14:52:05.065935 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ad6f98f8fa445ab2100f29a7ffb991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.066027 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_28a52958055546da964198b7c17f8b8c" } modify_external_catalogs: true
I20250115 14:52:05.066809 1167 raft_consensus.cc:2238] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:05.067116 1167 raft_consensus.cc:2267] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.067622 1167 ts_tablet_manager.cc:1905] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.068413 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_944d3194b69f4c6a9c0899a204051442"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.068907 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_944d3194b69f4c6a9c0899a204051442 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.069957 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8a0954ff11744933ba4752116830cdd7" } modify_external_catalogs: true
I20250115 14:52:05.070119 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8a0954ff11744933ba4752116830cdd7" } modify_external_catalogs: true
I20250115 14:52:05.071035 1751 log.cc:1198] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/50273a0019bb4cd8977f7ef014372d85
I20250115 14:52:05.071324 1751 ts_tablet_manager.cc:1939] T 50273a0019bb4cd8977f7ef014372d85 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:05.072273 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 50273a0019bb4cd8977f7ef014372d85 (table default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]) successfully deleted
I20250115 14:52:05.073333 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 23ad6f98f8fa445ab2100f29a7ffb991 (DEFAULT_TABLE table=default.Table_29e435c06be34d20b3fe6ab2745835d8 [id=7caed57491e248ee8f5ac239666c8a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.073657 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ad6f98f8fa445ab2100f29a7ffb991. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.075294 893 tablet_service.cc:1467] Processing CreateTablet for tablet 35301651222249d0b3439e0638f599f2 (DEFAULT_TABLE table=default.Table_44bc726782ff4d8d8a341acf255a4dd8 [id=190589a22d424365a7f858a214fd0f5b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:05.075867 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 288b0e455a854464940f65a2ab909d0b (table default.Table_61edbe7db3b44d02a60d60d865a86ec5 [id=5ae3104eeac943f69d0c1d79e148ab89], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 288b0e455a854464940f65a2ab909d0b consensus state has no leader
I20250115 14:52:05.076014 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35301651222249d0b3439e0638f599f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.076258 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ab3fe6097891488eb31670fe723ef7d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.076861 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab3fe6097891488eb31670fe723ef7d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.078883 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9118acb2bd5e43e9b5d4b0d2379fadfb" } modify_external_catalogs: true
I20250115 14:52:05.079068 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9118acb2bd5e43e9b5d4b0d2379fadfb" } modify_external_catalogs: true
I20250115 14:52:05.079315 743 tablet_service.cc:1467] Processing CreateTablet for tablet 35301651222249d0b3439e0638f599f2 (DEFAULT_TABLE table=default.Table_44bc726782ff4d8d8a341acf255a4dd8 [id=190589a22d424365a7f858a214fd0f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.079646 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35301651222249d0b3439e0638f599f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.088398 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet e034d7761a5247c79414b7112a1c97b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.090633 1876 raft_consensus.cc:491] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.090754 1876 raft_consensus.cc:513] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.091321 1876 leader_election.cc:290] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:05.091393 1168 tablet_bootstrap.cc:492] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.091507 1168 ts_tablet_manager.cc:1397] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20250115 14:52:05.092006 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e034d7761a5247c79414b7112a1c97b2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:05.092137 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
I20250115 14:52:05.092154 1168 raft_consensus.cc:357] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.092336 1168 raft_consensus.cc:383] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.092406 1168 raft_consensus.cc:738] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
W20250115 14:52:05.092711 984 leader_election.cc:343] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.092621 1168 consensus_queue.cc:260] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.093353 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e034d7761a5247c79414b7112a1c97b2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:05.094017 983 leader_election.cc:343] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.094110 983 leader_election.cc:304] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.096406 1876 raft_consensus.cc:2743] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.097584 1167 ts_tablet_manager.cc:1918] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:05.097800 1167 log.cc:1198] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e9e5bc6131944fcab9c550caac4f8d85
I20250115 14:52:05.098364 1167 ts_tablet_manager.cc:1939] T e9e5bc6131944fcab9c550caac4f8d85 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:05.102807 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 611) took 1900 ms. Trace:
I20250115 14:52:05.102943 1167 rpcz_store.cc:276] 0115 14:52:03.202079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.202134 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:05.058383 (+1856249us) ts_tablet_manager.cc:1145] Deleting tablet e9e5bc6131944fcab9c550caac4f8d85
0115 14:52:05.058392 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.077867 (+ 19475us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.097451 (+ 19584us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.102787 (+ 5336us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1854442}
I20250115 14:52:05.103440 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet e9e5bc6131944fcab9c550caac4f8d85 (table default.Table_609197916c9b4c81aebdaf70e78cc436 [id=33e3df28f2d445c5ba4e8b0ce4458c1a]) successfully deleted
I20250115 14:52:05.103875 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 612) took 1862 ms. Trace:
I20250115 14:52:05.103991 1167 rpcz_store.cc:276] 0115 14:52:03.241097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.241150 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:05.103753 (+1862603us) ts_tablet_manager.cc:1145] Deleting tablet f3280e8ba6db4446aa3e3bfe2836abfa
0115 14:52:05.103762 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.103860 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1862430}
W20250115 14:52:05.104578 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f3280e8ba6db4446aa3e3bfe2836abfa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3280e8ba6db4446aa3e3bfe2836abfa already in progress: creating tablet
I20250115 14:52:05.105391 1167 tablet_replica.cc:331] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:05.105976 1167 raft_consensus.cc:2238] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.108477 1167 raft_consensus.cc:2267] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.108775 446 master_service.cc:946] ReplaceTablet: received request to replace tablet be8e12ccb8d14c1c9258dc3f81d1527d from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.112726 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet dff155a5c2b0479a90e74993de6f402d (table default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]). Expected version 1 got 0
I20250115 14:52:05.112938 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
I20250115 14:52:05.113860 911 raft_consensus.cc:1270] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer d2d7ecece2ab49e1be7079216517d13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:05.114522 2110 consensus_queue.cc:1035] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:05.115335 2013 raft_consensus.cc:491] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.115489 1049 raft_consensus.cc:1270] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer d2d7ecece2ab49e1be7079216517d13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:05.115454 2013 raft_consensus.cc:513] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.116071 2013 leader_election.cc:290] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.117120 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf94077e0e343b8938b081102dfd2af" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:05.117818 848 leader_election.cc:343] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.117946 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf94077e0e343b8938b081102dfd2af" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.118119 1049 raft_consensus.cc:2463] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:05.118249 1167 ts_tablet_manager.cc:1905] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.118515 847 leader_election.cc:304] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.118506 2110 consensus_queue.cc:1035] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:05.118872 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d8fd4b26294e8ca6745b95f2e03231 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.118925 2013 raft_consensus.cc:2798] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:05.118999 2013 raft_consensus.cc:491] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:05.119055 2013 raft_consensus.cc:3054] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.119089 1166 tablet_replica.cc:331] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:05.119246 1166 raft_consensus.cc:2238] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.119369 1166 raft_consensus.cc:2267] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.120008 1166 ts_tablet_manager.cc:1905] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.121652 2013 raft_consensus.cc:513] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.122380 2013 leader_election.cc:290] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.122752 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf94077e0e343b8938b081102dfd2af" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:05.123131 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf94077e0e343b8938b081102dfd2af" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:05.123283 1048 raft_consensus.cc:3054] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.125653 1048 raft_consensus.cc:2463] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:05.127169 2013 raft_consensus.cc:491] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.127274 2013 raft_consensus.cc:513] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.127656 1166 ts_tablet_manager.cc:1918] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.127758 1166 log.cc:1198] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/59d8fd4b26294e8ca6745b95f2e03231
I20250115 14:52:05.128055 2013 leader_election.cc:290] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.131244 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet fb81e8fcc38a41e69cfb0e34dbaaac84
I20250115 14:52:05.131693 452 catalog_manager.cc:6596] ReplaceTablet: tablet fb81e8fcc38a41e69cfb0e34dbaaac84 deleted and replaced by tablet d7558b4d7453453f83facb0fc7e371e8
I20250115 14:52:05.131822 1166 ts_tablet_manager.cc:1939] T 59d8fd4b26294e8ca6745b95f2e03231 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:05.132671 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d1259256b7744e739581125611abbe67
I20250115 14:52:05.134276 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 65a6f83730f74d13823bd15d93538df7
I20250115 14:52:05.134941 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 57aecd22f97c41618f90d14c2f53a7c9
I20250115 14:52:05.137068 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 57aecd22f97c41618f90d14c2f53a7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.137255 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 65a6f83730f74d13823bd15d93538df7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.138278 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_44bc726782ff4d8d8a341acf255a4dd8" } new_table_name: "default.Table_380c0dd6df914022a741ea83ed384911" modify_external_catalogs: true
I20250115 14:52:05.139813 879 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81e8fcc38a41e69cfb0e34dbaaac84 with delete_type TABLET_DATA_DELETED (replaced by tablet d7558b4d7453453f83facb0fc7e371e8) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.139765 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d3380e50077a425fbd4ded35b60ba49c" } new_table_name: "default.Table_e8f8acbb30db4f019ebeadda783b434a" modify_external_catalogs: true
I20250115 14:52:05.139995 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d1259256b7744e739581125611abbe67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.140110 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 65a6f83730f74d13823bd15d93538df7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.140210 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 57aecd22f97c41618f90d14c2f53a7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.140800 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet d1259256b7744e739581125611abbe67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.137439 1168 ts_tablet_manager.cc:1428] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20250115 14:52:05.141029 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81e8fcc38a41e69cfb0e34dbaaac84 with delete_type TABLET_DATA_DELETED (replaced by tablet d7558b4d7453453f83facb0fc7e371e8) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.141013 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9c676839fdd84942afad4ab037b135ad" } new_table_name: "default.Table_83379c7aa94244638437e340ad9da9c5" modify_external_catalogs: true
I20250115 14:52:05.141330 1168 tablet_bootstrap.cc:492] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:05.142267 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4ab29505c6634887aa99118862eccf5d" } modify_external_catalogs: true
I20250115 14:52:05.142427 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4ab29505c6634887aa99118862eccf5d" } modify_external_catalogs: true
I20250115 14:52:05.144482 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 65a6f83730f74d13823bd15d93538df7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.144666 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa1827366a34f039fd12deb855f07b6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:05.144680 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 57aecd22f97c41618f90d14c2f53a7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:05.145290 848 leader_election.cc:343] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:05.145507 848 leader_election.cc:343] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.145751 847 leader_election.cc:304] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.148370 2013 raft_consensus.cc:2798] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:05.148622 2013 raft_consensus.cc:695] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:05.148938 2013 consensus_queue.cc:237] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.149119 1168 tablet_bootstrap.cc:654] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:05.150151 847 leader_election.cc:343] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: caa1827366a34f039fd12deb855f07b6
I20250115 14:52:05.150250 847 leader_election.cc:304] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.150561 1843 raft_consensus.cc:2743] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.150887 741 tablet_service.cc:1514] Processing DeleteTablet for tablet d1259256b7744e739581125611abbe67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.151346 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_84124eeb8c424548bd8b58245bb7823f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.151940 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84124eeb8c424548bd8b58245bb7823f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.153784 755 tablet_service.cc:1514] Processing DeleteTablet for tablet fb81e8fcc38a41e69cfb0e34dbaaac84 with delete_type TABLET_DATA_DELETED (replaced by tablet d7558b4d7453453f83facb0fc7e371e8) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.154582 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7f8c179068074379b0e5baec32f2699b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.155028 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f8c179068074379b0e5baec32f2699b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.158051 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet be8e12ccb8d14c1c9258dc3f81d1527d
I20250115 14:52:05.158516 446 catalog_manager.cc:6596] ReplaceTablet: tablet be8e12ccb8d14c1c9258dc3f81d1527d deleted and replaced by tablet eba5819df2944a84a636c6934b18510d
I20250115 14:52:05.159307 889 tablet_service.cc:1514] Processing DeleteTablet for tablet be8e12ccb8d14c1c9258dc3f81d1527d with delete_type TABLET_DATA_DELETED (replaced by tablet eba5819df2944a84a636c6934b18510d) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.160109 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 59d8fd4b26294e8ca6745b95f2e03231 (table default.Table_98cb517939064897baf89d74e6e0355e [id=757cda13648a4e3d9b02d70555966106]) successfully deleted
W20250115 14:52:05.160413 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 65a6f83730f74d13823bd15d93538df7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65a6f83730f74d13823bd15d93538df7 already in progress: creating tablet
I20250115 14:52:05.160616 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet be8e12ccb8d14c1c9258dc3f81d1527d with delete_type TABLET_DATA_DELETED (replaced by tablet eba5819df2944a84a636c6934b18510d) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:05.161157 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ca5cc55b50ee4953800f10efe0ffb546 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca5cc55b50ee4953800f10efe0ffb546 already in progress: creating tablet
W20250115 14:52:05.161365 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 57aecd22f97c41618f90d14c2f53a7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57aecd22f97c41618f90d14c2f53a7c9 already in progress: creating tablet
W20250115 14:52:05.161610 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d1259256b7744e739581125611abbe67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1259256b7744e739581125611abbe67 already in progress: creating tablet
W20250115 14:52:05.161921 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet fb81e8fcc38a41e69cfb0e34dbaaac84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb81e8fcc38a41e69cfb0e34dbaaac84 already in progress: creating tablet
I20250115 14:52:05.161917 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 35301651222249d0b3439e0638f599f2 (DEFAULT_TABLE table=default.Table_44bc726782ff4d8d8a341acf255a4dd8 [id=190589a22d424365a7f858a214fd0f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.162323 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35301651222249d0b3439e0638f599f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.162421 755 tablet_service.cc:1514] Processing DeleteTablet for tablet be8e12ccb8d14c1c9258dc3f81d1527d with delete_type TABLET_DATA_DELETED (replaced by tablet eba5819df2944a84a636c6934b18510d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.164042 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_a08bf8546b9148dc97c582984b4409c8 [id=7628dbb6818f443680a97fef665d6d2b]). Expected version 3 got 0
W20250115 14:52:05.168236 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 48b15de9bf8943cdb9e5abd943484993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48b15de9bf8943cdb9e5abd943484993 already in progress: creating tablet
I20250115 14:52:05.169333 1751 tablet_replica.cc:331] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:05.169697 1751 raft_consensus.cc:2238] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.170138 1751 raft_consensus.cc:2267] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.170532 1168 tablet_bootstrap.cc:492] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.170897 1751 ts_tablet_manager.cc:1905] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.172327 1168 ts_tablet_manager.cc:1397] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250115 14:52:05.173583 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
I20250115 14:52:05.173913 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_a08bf8546b9148dc97c582984b4409c8 [id=7628dbb6818f443680a97fef665d6d2b]). Expected version 3 got 0
I20250115 14:52:05.175235 1016 tablet_service.cc:1467] Processing CreateTablet for tablet a739d36ef9d54c089667ff32de875af6 (DEFAULT_TABLE table=default.Table_ec9e96994cca43569c5976b968296550 [id=18a17a01a698428a8d6ed94b722b3175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.175608 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a739d36ef9d54c089667ff32de875af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.177110 2131 tablet.cc:1721] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:05.180397 2149 tablet.cc:1721] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:05.182852 889 tablet_service.cc:1467] Processing CreateTablet for tablet dadeb51154c947209dc1e00ebf844b74 (DEFAULT_TABLE table=default.Table_5c0d0b9700e4413d85d5a1dbc9a23a45 [id=e024c299d4b049ef857bf13773b76e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.183017 893 tablet_service.cc:1467] Processing CreateTablet for tablet a739d36ef9d54c089667ff32de875af6 (DEFAULT_TABLE table=default.Table_ec9e96994cca43569c5976b968296550 [id=18a17a01a698428a8d6ed94b722b3175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.183940 1016 tablet_service.cc:1467] Processing CreateTablet for tablet dadeb51154c947209dc1e00ebf844b74 (DEFAULT_TABLE table=default.Table_5c0d0b9700e4413d85d5a1dbc9a23a45 [id=e024c299d4b049ef857bf13773b76e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.184252 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dadeb51154c947209dc1e00ebf844b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.187281 883 tablet_service.cc:1467] Processing CreateTablet for tablet e3c54fdc672f4709a44b2542212dfb5e (DEFAULT_TABLE table=default.Table_32080fc25c67476db569c3357f008e2c [id=9b453c31d5cd4e6a805d95bd66faa8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.192615 1025 tablet_service.cc:1467] Processing CreateTablet for tablet e3c54fdc672f4709a44b2542212dfb5e (DEFAULT_TABLE table=default.Table_32080fc25c67476db569c3357f008e2c [id=9b453c31d5cd4e6a805d95bd66faa8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.192950 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c54fdc672f4709a44b2542212dfb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.192982 749 tablet_service.cc:1467] Processing CreateTablet for tablet e3c54fdc672f4709a44b2542212dfb5e (DEFAULT_TABLE table=default.Table_32080fc25c67476db569c3357f008e2c [id=9b453c31d5cd4e6a805d95bd66faa8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.193288 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c54fdc672f4709a44b2542212dfb5e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.198030 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35301651222249d0b3439e0638f599f2 consensus state has no leader
W20250115 14:52:05.198532 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64994f0d3fdd43e2935786bf11544fb6 consensus state has no leader
I20250115 14:52:05.199792 1162 tablet_bootstrap.cc:492] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
W20250115 14:52:05.199949 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efb8b001dd66456d9e49e9e8a6848ccf consensus state has no leader
I20250115 14:52:05.199913 1162 ts_tablet_manager.cc:1397] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.209s user 0.003s sys 0.008s
I20250115 14:52:05.200330 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 567365040b6d4d6eba04112c8d4c3dfb
I20250115 14:52:05.201210 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 567365040b6d4d6eba04112c8d4c3dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.201426 1162 raft_consensus.cc:357] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.201588 1162 raft_consensus.cc:383] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.201632 1162 raft_consensus.cc:738] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.201828 1162 consensus_queue.cc:260] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.202296 1162 ts_tablet_manager.cc:1428] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:05.202526 1162 tablet_bootstrap.cc:492] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:05.202452 2079 tablet.cc:1721] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:05.203020 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:05.203650 1167 ts_tablet_manager.cc:1918] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:05.204001 1162 tablet_bootstrap.cc:654] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.204386 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 567365040b6d4d6eba04112c8d4c3dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.205305 741 tablet_service.cc:1467] Processing CreateTablet for tablet dadeb51154c947209dc1e00ebf844b74 (DEFAULT_TABLE table=default.Table_5c0d0b9700e4413d85d5a1dbc9a23a45 [id=e024c299d4b049ef857bf13773b76e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.205638 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dadeb51154c947209dc1e00ebf844b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.206319 1025 tablet_service.cc:1467] Processing CreateTablet for tablet cee072daff2d4943b68261806666e4a0 (DEFAULT_TABLE table=default.Table_5f4c7f8a9cd745ea935c1997f991a1f1 [id=e64b65f9e24e4223a8dd6f688478ec87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.205505 880 tablet_service.cc:1467] Processing CreateTablet for tablet cee072daff2d4943b68261806666e4a0 (DEFAULT_TABLE table=default.Table_5f4c7f8a9cd745ea935c1997f991a1f1 [id=e64b65f9e24e4223a8dd6f688478ec87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.206601 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee072daff2d4943b68261806666e4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.206794 1167 log.cc:1198] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/7e3ec000089a47a2930dec382192cd6f
I20250115 14:52:05.210424 1167 ts_tablet_manager.cc:1939] T 7e3ec000089a47a2930dec382192cd6f P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:05.210626 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_a08bf8546b9148dc97c582984b4409c8 [id=7628dbb6818f443680a97fef665d6d2b]). Expected version 3 got 0
I20250115 14:52:05.211750 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9339e2f34b994dd7a9f6385ee32f1b05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.212365 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9339e2f34b994dd7a9f6385ee32f1b05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.214936 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_aa6a106222694bbfbe6c26b0f8e65396" } new_table_name: "default.Table_11d56e79fc964adbb45d31451183c7d9" modify_external_catalogs: true
W20250115 14:52:05.216954 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet be8e12ccb8d14c1c9258dc3f81d1527d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be8e12ccb8d14c1c9258dc3f81d1527d already in progress: creating tablet
I20250115 14:52:05.217128 2117 raft_consensus.cc:491] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.217244 2117 raft_consensus.cc:513] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.217804 2117 leader_election.cc:290] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:05.218912 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff155a5c2b0479a90e74993de6f402d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:05.219487 911 raft_consensus.cc:3054] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.223047 2110 raft_consensus.cc:491] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.223156 2110 raft_consensus.cc:513] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.223659 2110 leader_election.cc:290] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.223785 2110 consensus_queue.cc:1035] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:05.225118 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffa997ca6c46559830b1dedf2d1a66" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.225317 751 tablet_service.cc:1467] Processing CreateTablet for tablet cee072daff2d4943b68261806666e4a0 (DEFAULT_TABLE table=default.Table_5f4c7f8a9cd745ea935c1997f991a1f1 [id=e64b65f9e24e4223a8dd6f688478ec87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.225654 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee072daff2d4943b68261806666e4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.228432 755 tablet_service.cc:1467] Processing CreateTablet for tablet a739d36ef9d54c089667ff32de875af6 (DEFAULT_TABLE table=default.Table_ec9e96994cca43569c5976b968296550 [id=18a17a01a698428a8d6ed94b722b3175]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.229061 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5ffa997ca6c46559830b1dedf2d1a66" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:05.229523 710 leader_election.cc:304] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:05.229884 2117 raft_consensus.cc:2743] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.230996 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 567365040b6d4d6eba04112c8d4c3dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.231627 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff155a5c2b0479a90e74993de6f402d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.232085 710 leader_election.cc:304] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:05.232317 2117 raft_consensus.cc:2743] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.233248 451 catalog_manager.cc:5526] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:05.233693 451 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet fbf94077e0e343b8938b081102dfd2af (table default.Table_bbd25ad975704fbcae339bf950fd1052 [id=00a89ba1b93f4132aae8fe6e748df77e]). Expected version 2 got 0
I20250115 14:52:05.234207 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dadeb51154c947209dc1e00ebf844b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.234210 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a739d36ef9d54c089667ff32de875af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.236173 751 tablet_service.cc:1467] Processing CreateTablet for tablet 36514c5399924e98b3bc38004a33b919 (DEFAULT_TABLE table=default.Table_ec36b5656e984a2a9468f7d11f75d663 [id=0991aca01b7a4542b941c28bf2d7dab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.236544 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36514c5399924e98b3bc38004a33b919. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.238196 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3c54fdc672f4709a44b2542212dfb5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.240756 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cee072daff2d4943b68261806666e4a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.241854 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_a08bf8546b9148dc97c582984b4409c8 [id=7628dbb6818f443680a97fef665d6d2b] alter complete (version 3)
W20250115 14:52:05.242993 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 567365040b6d4d6eba04112c8d4c3dfb because the tablet was not found. No further retry: Not found: Tablet not found: 567365040b6d4d6eba04112c8d4c3dfb
I20250115 14:52:05.244565 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a739d36ef9d54c089667ff32de875af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.244885 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_74edc08b682e4c5cb696e54fcadaf5f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.245882 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74edc08b682e4c5cb696e54fcadaf5f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.245920 2013 raft_consensus.cc:491] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.246157 2013 raft_consensus.cc:513] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.247258 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be63bae96c834a8bbbb1174c027935a2" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:05.247483 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 615) took 1788 ms. Trace:
W20250115 14:52:05.247749 848 leader_election.cc:343] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.247555 1167 rpcz_store.cc:276] 0115 14:52:03.458970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.459017 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:05.105348 (+1646331us) ts_tablet_manager.cc:1145] Deleting tablet 7e3ec000089a47a2930dec382192cd6f
0115 14:52:05.105357 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.176407 (+ 71050us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.203100 (+ 26693us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.247467 (+ 44367us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646161}
I20250115 14:52:05.248386 1167 tablet_replica.cc:331] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:05.248345 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 7e3ec000089a47a2930dec382192cd6f (table default.Table_86abf42208c1422c8cede2bf48d37c3a [id=5a3acfd8a17e40eaa8c529ba7e9bc64e]) successfully deleted
W20250115 14:52:05.248559 847 leader_election.cc:343] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: be63bae96c834a8bbbb1174c027935a2
I20250115 14:52:05.248636 1167 raft_consensus.cc:2238] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.248663 847 leader_election.cc:304] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.248806 1167 raft_consensus.cc:2267] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.248917 1843 raft_consensus.cc:2743] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.248909 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b28114e5556947c4aaba883427f90c2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:05.249320 1167 ts_tablet_manager.cc:1905] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:05.249365 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b28114e5556947c4aaba883427f90c2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.247942 2013 leader_election.cc:290] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.251485 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fcacad621d8e46eebb110daa55789690" } new_table_name: "default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec" modify_external_catalogs: true
I20250115 14:52:05.252483 880 tablet_service.cc:1467] Processing CreateTablet for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 (DEFAULT_TABLE table=default.Table_57cdcb46b45d430b84affd591c061ced [id=787923ba07604b4abf27cbb0c831995a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.252883 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.255156 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1c56d7edc9fa481db33c058487afde17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.255615 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c56d7edc9fa481db33c058487afde17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.259050 1167 ts_tablet_manager.cc:1918] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.259138 1167 log.cc:1198] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e198b4e1a65b4704bf2bb8f7d5987385
I20250115 14:52:05.259424 1167 ts_tablet_manager.cc:1939] T e198b4e1a65b4704bf2bb8f7d5987385 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:05.259660 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 614) took 1800 ms. Trace:
I20250115 14:52:05.259742 1167 rpcz_store.cc:276] 0115 14:52:03.458865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.459317 (+ 452us) service_pool.cc:225] Handling call
0115 14:52:05.248346 (+1789029us) ts_tablet_manager.cc:1145] Deleting tablet e198b4e1a65b4704bf2bb8f7d5987385
0115 14:52:05.248355 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.256095 (+ 7740us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.258977 (+ 2882us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.259647 (+ 670us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1788896}
I20250115 14:52:05.262651 1164 tablet_bootstrap.cc:492] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:05.263034 1164 ts_tablet_manager.cc:1397] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.212s user 0.004s sys 0.004s
I20250115 14:52:05.263571 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 36514c5399924e98b3bc38004a33b919 (DEFAULT_TABLE table=default.Table_ec36b5656e984a2a9468f7d11f75d663 [id=0991aca01b7a4542b941c28bf2d7dab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.263784 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:05.264161 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36514c5399924e98b3bc38004a33b919. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.267627 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 009ddfe9dfe14ba2b067f907a9d13538 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.269959 741 tablet_service.cc:1467] Processing CreateTablet for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 (DEFAULT_TABLE table=default.Table_57cdcb46b45d430b84affd591c061ced [id=787923ba07604b4abf27cbb0c831995a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.270315 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0b1e4ccdc6ce444195b8bf86c86ff87b" } modify_external_catalogs: true
I20250115 14:52:05.270499 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0b1e4ccdc6ce444195b8bf86c86ff87b" } modify_external_catalogs: true
I20250115 14:52:05.270692 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.265117 1164 raft_consensus.cc:357] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.272876 1168 raft_consensus.cc:357] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.273156 1164 raft_consensus.cc:383] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.273350 1164 raft_consensus.cc:738] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.274137 1025 tablet_service.cc:1467] Processing CreateTablet for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 (DEFAULT_TABLE table=default.Table_57cdcb46b45d430b84affd591c061ced [id=787923ba07604b4abf27cbb0c831995a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.274523 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.276268 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 43605c995df4400e91139199c9514bfd with delete_type TABLET_DATA_DELETED (replaced by tablet e910c387e6a24caf95e14b57c8b83e6f) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.274586 755 tablet_service.cc:1467] Processing CreateTablet for tablet 33193ebbb6e34f1785a1687f78ff1258 (DEFAULT_TABLE table=default.Table_ac31746e6f2f4855b46e06e0f7d95a16 [id=47a9b8aacf9b4a43bd33a947e29a5eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.276597 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33193ebbb6e34f1785a1687f78ff1258. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.278088 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea8ab6a34caa486ab929af2cac3ce957 consensus state has no leader
I20250115 14:52:05.283378 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 33193ebbb6e34f1785a1687f78ff1258 (DEFAULT_TABLE table=default.Table_ac31746e6f2f4855b46e06e0f7d95a16 [id=47a9b8aacf9b4a43bd33a947e29a5eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.283754 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33193ebbb6e34f1785a1687f78ff1258. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.284224 2059 log.cc:926] Time spent T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: Append to log took a long time: real 0.053s user 0.002s sys 0.001s
I20250115 14:52:05.282860 1164 consensus_queue.cc:260] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.293422 1164 ts_tablet_manager.cc:1428] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
W20250115 14:52:05.293588 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be2fd31f74a94787a5e8ccae3c5ef245 consensus state has no leader
I20250115 14:52:05.294178 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 009ddfe9dfe14ba2b067f907a9d13538
I20250115 14:52:05.294600 447 catalog_manager.cc:6596] ReplaceTablet: tablet 009ddfe9dfe14ba2b067f907a9d13538 deleted and replaced by tablet ee5671e8a08d47f7bea9fc6af2f53b2d
I20250115 14:52:05.295370 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet db5518ac647c472bb47ed741022c820a
I20250115 14:52:05.295576 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 009ddfe9dfe14ba2b067f907a9d13538 with delete_type TABLET_DATA_DELETED (replaced by tablet ee5671e8a08d47f7bea9fc6af2f53b2d) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.296463 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet db5518ac647c472bb47ed741022c820a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.297313 1751 ts_tablet_manager.cc:1918] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.298476 2110 consensus_queue.cc:1035] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:05.299665 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7baacab95e6c4166b3437a9448306088 (DEFAULT_TABLE table=default.Table_1bef4c655e464405a4b2893dba5c9eca [id=59d03b12bb96413e8f7e7627b977a670]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.300026 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7baacab95e6c4166b3437a9448306088. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.297930 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 7baacab95e6c4166b3437a9448306088 (DEFAULT_TABLE table=default.Table_1bef4c655e464405a4b2893dba5c9eca [id=59d03b12bb96413e8f7e7627b977a670]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.300958 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7baacab95e6c4166b3437a9448306088. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.300616 1162 tablet_bootstrap.cc:492] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:05.304450 1162 ts_tablet_manager.cc:1397] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.102s user 0.001s sys 0.002s
I20250115 14:52:05.305222 1162 raft_consensus.cc:357] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.305466 1162 raft_consensus.cc:383] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.305559 1162 raft_consensus.cc:738] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.305837 1162 consensus_queue.cc:260] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.306367 1162 ts_tablet_manager.cc:1428] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:05.306641 1162 tablet_bootstrap.cc:492] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:05.308519 1162 tablet_bootstrap.cc:654] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.311448 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 009ddfe9dfe14ba2b067f907a9d13538 with delete_type TABLET_DATA_DELETED (replaced by tablet ee5671e8a08d47f7bea9fc6af2f53b2d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.273526 883 tablet_service.cc:1467] Processing CreateTablet for tablet 33193ebbb6e34f1785a1687f78ff1258 (DEFAULT_TABLE table=default.Table_ac31746e6f2f4855b46e06e0f7d95a16 [id=47a9b8aacf9b4a43bd33a947e29a5eca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.312381 1168 raft_consensus.cc:383] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.312450 1168 raft_consensus.cc:738] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.312609 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33193ebbb6e34f1785a1687f78ff1258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.312636 1168 consensus_queue.cc:260] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.313104 749 tablet_service.cc:1514] Processing DeleteTablet for tablet db5518ac647c472bb47ed741022c820a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.314834 749 tablet_service.cc:1467] Processing CreateTablet for tablet fbf5da59db79418498db09ec8ea3ac0a (DEFAULT_TABLE table=default.Table_4ced70cc9c3d4302b3856c7dc634d096 [id=103e5591196f4fb2a5823c71c0fe7ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.315165 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbf5da59db79418498db09ec8ea3ac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.276235 893 tablet_service.cc:1467] Processing CreateTablet for tablet 36514c5399924e98b3bc38004a33b919 (DEFAULT_TABLE table=default.Table_ec36b5656e984a2a9468f7d11f75d663 [id=0991aca01b7a4542b941c28bf2d7dab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.318801 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36514c5399924e98b3bc38004a33b919. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.321151 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 009ddfe9dfe14ba2b067f907a9d13538 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 009ddfe9dfe14ba2b067f907a9d13538 already in progress: creating tablet
I20250115 14:52:05.292582 889 tablet_service.cc:1467] Processing CreateTablet for tablet 7baacab95e6c4166b3437a9448306088 (DEFAULT_TABLE table=default.Table_1bef4c655e464405a4b2893dba5c9eca [id=59d03b12bb96413e8f7e7627b977a670]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.295245 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 009ddfe9dfe14ba2b067f907a9d13538 with delete_type TABLET_DATA_DELETED (replaced by tablet ee5671e8a08d47f7bea9fc6af2f53b2d) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.296083 886 tablet_service.cc:1514] Processing DeleteTablet for tablet db5518ac647c472bb47ed741022c820a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.298925 885 tablet_service.cc:1467] Processing CreateTablet for tablet fbf5da59db79418498db09ec8ea3ac0a (DEFAULT_TABLE table=default.Table_4ced70cc9c3d4302b3856c7dc634d096 [id=103e5591196f4fb2a5823c71c0fe7ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.325860 1164 tablet_bootstrap.cc:492] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.322321 2138 raft_consensus.cc:491] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.326753 2138 raft_consensus.cc:513] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.327368 2138 leader_election.cc:290] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.322391 2083 raft_consensus.cc:491] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.327546 2083 raft_consensus.cc:513] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.328032 2083 leader_election.cc:290] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.328217 1164 tablet_bootstrap.cc:654] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:05.329723 847 leader_election.cc:343] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 010e6535b6bc44f391e6f2b6f1686532
I20250115 14:52:05.329826 1168 ts_tablet_manager.cc:1428] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.157s user 0.002s sys 0.000s
I20250115 14:52:05.331521 886 tablet_service.cc:1467] Processing CreateTablet for tablet 9aefb12a03784eb895a17fd8c571aa24 (DEFAULT_TABLE table=default.Table_fa4c201ba84d4cf5902269c4c6e4f50c [id=bb1ad05767ba455e8d3c971c6557b498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.332540 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbf5da59db79418498db09ec8ea3ac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.321972 2013 raft_consensus.cc:491] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.332707 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aefb12a03784eb895a17fd8c571aa24. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.333927 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet db5518ac647c472bb47ed741022c820a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db5518ac647c472bb47ed741022c820a already in progress: creating tablet
I20250115 14:52:05.332878 2013 raft_consensus.cc:513] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.336529 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7baacab95e6c4166b3437a9448306088. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.337893 847 leader_election.cc:343] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 5bc975b044dd439dbd1f8f9a82cce751
I20250115 14:52:05.338099 2013 leader_election.cc:290] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.344122 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5ea7727ee96e426594a8c8af14d9545d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.344696 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ea7727ee96e426594a8c8af14d9545d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.345897 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d" } modify_external_catalogs: true
I20250115 14:52:05.346092 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d" } modify_external_catalogs: true
I20250115 14:52:05.347434 1168 tablet_bootstrap.cc:492] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:05.347309 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9da42aab96b54c6f84d977c95677d7e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.347802 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9da42aab96b54c6f84d977c95677d7e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.350690 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ec36b5656e984a2a9468f7d11f75d663" } modify_external_catalogs: true
I20250115 14:52:05.350860 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ec36b5656e984a2a9468f7d11f75d663" } modify_external_catalogs: true
I20250115 14:52:05.330279 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338c0f6f39fd4335bbcda733b28b8bf8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:05.330490 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010e6535b6bc44f391e6f2b6f1686532" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:05.351948 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c72e28daf2c44e519f2e934f811954e1" } modify_external_catalogs: true
I20250115 14:52:05.352104 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c72e28daf2c44e519f2e934f811954e1" } modify_external_catalogs: true
I20250115 14:52:05.338543 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc975b044dd439dbd1f8f9a82cce751" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:05.341336 745 tablet_service.cc:1467] Processing CreateTablet for tablet e59b36e986e146b9980c1f71599f32aa (DEFAULT_TABLE table=default.Table_8be6e58a536c49c5add0072a13fa8be5 [id=56023f3f9346423595a409d330037dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.352622 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59b36e986e146b9980c1f71599f32aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.353205 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d42cd252db384a70b6ef1a2595274afa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.353621 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d42cd252db384a70b6ef1a2595274afa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.354563 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d43d582508924d259b43a911c5d2c876"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.354984 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d43d582508924d259b43a911c5d2c876 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.356165 1167 tablet_replica.cc:331] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:05.342368 739 tablet_service.cc:1467] Processing CreateTablet for tablet 9aefb12a03784eb895a17fd8c571aa24 (DEFAULT_TABLE table=default.Table_fa4c201ba84d4cf5902269c4c6e4f50c [id=bb1ad05767ba455e8d3c971c6557b498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.356366 1167 raft_consensus.cc:2238] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.356501 1167 raft_consensus.cc:2267] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.356540 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aefb12a03784eb895a17fd8c571aa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.356981 1167 ts_tablet_manager.cc:1905] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.357234 2013 raft_consensus.cc:491] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.357338 2013 raft_consensus.cc:513] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.358052 2013 leader_election.cc:290] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.360546 893 tablet_service.cc:1467] Processing CreateTablet for tablet e59b36e986e146b9980c1f71599f32aa (DEFAULT_TABLE table=default.Table_8be6e58a536c49c5add0072a13fa8be5 [id=56023f3f9346423595a409d330037dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.360926 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59b36e986e146b9980c1f71599f32aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.362313 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f793828f2ca49639c57849afc61ce9e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:05.364548 848 leader_election.cc:343] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:05.364756 848 leader_election.cc:343] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.364835 848 leader_election.cc:304] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:05.365226 848 leader_election.cc:343] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.365317 848 leader_election.cc:304] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.365610 2083 raft_consensus.cc:491] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.365734 2083 raft_consensus.cc:513] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.366252 2083 leader_election.cc:290] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.366364 2083 raft_consensus.cc:2743] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.366497 2083 raft_consensus.cc:2743] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:05.367375 847 leader_election.cc:343] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 37ad2b7350294375bd30fb4b46906491
I20250115 14:52:05.343328 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ad2b7350294375bd30fb4b46906491 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.367652 1166 tablet_replica.cc:331] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:05.367830 1166 raft_consensus.cc:2238] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.367990 1166 raft_consensus.cc:2267] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.367964 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ad2b7350294375bd30fb4b46906491" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:05.368396 848 leader_election.cc:343] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.368733 1166 ts_tablet_manager.cc:1905] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:05.368732 848 leader_election.cc:343] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.368819 848 leader_election.cc:304] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.369004 2138 raft_consensus.cc:2743] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.311430 1751 log.cc:1198] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/01291d41a2b345b4bfd00e6be0fb0129
I20250115 14:52:05.372642 1751 ts_tablet_manager.cc:1939] T 01291d41a2b345b4bfd00e6be0fb0129 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:05.373281 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 01291d41a2b345b4bfd00e6be0fb0129 (table default.Table_21fa1a9206d64c8bb19a405f37518a65 [id=e31104604e8648c08fd706e44150cda1]) successfully deleted
I20250115 14:52:05.374372 1166 ts_tablet_manager.cc:1918] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.374480 1166 log.cc:1198] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/37ad2b7350294375bd30fb4b46906491
I20250115 14:52:05.374758 1166 ts_tablet_manager.cc:1939] T 37ad2b7350294375bd30fb4b46906491 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:05.384605 1168 tablet_bootstrap.cc:654] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.384645 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 37ad2b7350294375bd30fb4b46906491 (table default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]) successfully deleted
I20250115 14:52:05.327953 1019 tablet_service.cc:1467] Processing CreateTablet for tablet fbf5da59db79418498db09ec8ea3ac0a (DEFAULT_TABLE table=default.Table_4ced70cc9c3d4302b3856c7dc634d096 [id=103e5591196f4fb2a5823c71c0fe7ee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.329193 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338c0f6f39fd4335bbcda733b28b8bf8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.393071 1048 raft_consensus.cc:2463] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:05.393464 847 leader_election.cc:304] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.394480 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbf5da59db79418498db09ec8ea3ac0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.394630 2013 raft_consensus.cc:2798] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:05.394752 2013 raft_consensus.cc:491] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:05.394860 2013 raft_consensus.cc:3054] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.396539 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 742792be08444792892bd9cd7336dda3 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.396981 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ad2b7350294375bd30fb4b46906491 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.397720 2013 raft_consensus.cc:513] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.398722 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338c0f6f39fd4335bbcda733b28b8bf8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:05.399001 2013 leader_election.cc:290] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:05.399180 848 leader_election.cc:343] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.361642 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f793828f2ca49639c57849afc61ce9e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.400656 1047 raft_consensus.cc:2463] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:05.398805 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338c0f6f39fd4335bbcda733b28b8bf8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:05.400954 1048 raft_consensus.cc:3054] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.401639 1168 tablet_bootstrap.cc:492] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.401743 1168 ts_tablet_manager.cc:1397] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20250115 14:52:05.403337 1048 raft_consensus.cc:2463] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:05.403676 847 leader_election.cc:304] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.404054 847 leader_election.cc:304] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.404402 2138 raft_consensus.cc:2798] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:05.404466 2138 raft_consensus.cc:491] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:05.404520 2138 raft_consensus.cc:3054] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.407152 2138 raft_consensus.cc:513] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.408038 2138 leader_election.cc:290] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.408053 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f793828f2ca49639c57849afc61ce9e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:05.408166 2138 raft_consensus.cc:2798] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:52:05.408586 848 leader_election.cc:343] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.408756 2138 raft_consensus.cc:695] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:05.408989 2138 consensus_queue.cc:237] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.409660 1168 raft_consensus.cc:357] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.409839 1168 raft_consensus.cc:383] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.409889 1168 raft_consensus.cc:738] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.411119 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f793828f2ca49639c57849afc61ce9e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:05.411284 1048 raft_consensus.cc:3054] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:05.412382 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 83415f1c07ad495cabd287bdcd54dad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83415f1c07ad495cabd287bdcd54dad4 already in progress: creating tablet
W20250115 14:52:05.412674 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a48a0af9422f43cf86531665c0fa1aee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48a0af9422f43cf86531665c0fa1aee already in progress: creating tablet
W20250115 14:52:05.412904 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet cc9ce226d3ba44d6908257bbd0df3a12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9ce226d3ba44d6908257bbd0df3a12 already in progress: creating tablet
W20250115 14:52:05.413069 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 37ad2b7350294375bd30fb4b46906491 because the tablet was not found. No further retry: Not found: Tablet not found: 37ad2b7350294375bd30fb4b46906491
W20250115 14:52:05.413111 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 already in progress: creating tablet
I20250115 14:52:05.413225 1168 consensus_queue.cc:260] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.414566 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_4b79de2dd7fe493f84a77ae3c477e2c8 [id=471b029f1dba4dd087deb339bba02013]). Expected version 1 got 0
I20250115 14:52:05.415786 1048 raft_consensus.cc:2463] T 1f793828f2ca49639c57849afc61ce9e P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:05.415916 454 master_service.cc:946] ReplaceTablet: received request to replace tablet dff155a5c2b0479a90e74993de6f402d from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.416209 847 leader_election.cc:304] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.416601 2083 raft_consensus.cc:2798] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:05.416714 2083 raft_consensus.cc:695] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:05.416916 2083 consensus_queue.cc:237] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.419811 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a600fa0f0ec049dca2803bf7d2c74a95
I20250115 14:52:05.420552 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_253ac9c181914d0f891baf1f8722dbd0" } new_table_name: "default.Table_526b537d21ae4762b6881e46346414c6" modify_external_catalogs: true
I20250115 14:52:05.420815 739 tablet_service.cc:1514] Processing DeleteTablet for tablet a600fa0f0ec049dca2803bf7d2c74a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:05.421443 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a600fa0f0ec049dca2803bf7d2c74a95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a600fa0f0ec049dca2803bf7d2c74a95 already in progress: creating tablet
I20250115 14:52:05.422623 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_35aa379f66554bac80e34f11e41062c7" } modify_external_catalogs: true
I20250115 14:52:05.422775 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_35aa379f66554bac80e34f11e41062c7" } modify_external_catalogs: true
I20250115 14:52:05.346030 1016 tablet_service.cc:1467] Processing CreateTablet for tablet e59b36e986e146b9980c1f71599f32aa (DEFAULT_TABLE table=default.Table_8be6e58a536c49c5add0072a13fa8be5 [id=56023f3f9346423595a409d330037dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.426329 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 36514c5399924e98b3bc38004a33b919
I20250115 14:52:05.426909 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 36514c5399924e98b3bc38004a33b919 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:05.427985 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 36514c5399924e98b3bc38004a33b919 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36514c5399924e98b3bc38004a33b919 already in progress: creating tablet
I20250115 14:52:05.428138 455 catalog_manager.cc:5526] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:05.428560 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 338c0f6f39fd4335bbcda733b28b8bf8 (table default.Table_f367ac1a82704324a73f7079165a6e45 [id=6756cde43edd4d1e8a2768c101c4aa19]). Expected version 3 got 0
I20250115 14:52:05.428578 2138 consensus_queue.cc:1035] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:05.429724 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e9febafb7e354e289cd3f864311f14cb
I20250115 14:52:05.429720 739 tablet_service.cc:1467] Processing CreateTablet for tablet 225d3c8c4c8c4f398ea99c4c32a67b57 (DEFAULT_TABLE table=default.Table_e215d4b1143c4ea49dde3acc698b7e60 [id=692f1b22ecbe4dc880e7757db8e7e02e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.430104 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225d3c8c4c8c4f398ea99c4c32a67b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.400525 1751 tablet_replica.cc:331] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:05.430932 1751 raft_consensus.cc:2238] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.430941 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d03b53f6993c4f3aae49c2155890bfc0" } new_table_name: "default.Table_ed3d35aa7cd046848bbe7989fdfbdae5" modify_external_catalogs: true
I20250115 14:52:05.431100 1751 raft_consensus.cc:2267] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.432139 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_40e515dbe043466f8d1ca690f14c6f42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.432610 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40e515dbe043466f8d1ca690f14c6f42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.433985 1751 ts_tablet_manager.cc:1905] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.434118 749 tablet_service.cc:1514] Processing DeleteTablet for tablet e9febafb7e354e289cd3f864311f14cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.434062 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_799a6df4f73a45c9a57ced3b6a102091"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.434521 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_799a6df4f73a45c9a57ced3b6a102091 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.434495 749 tablet_service.cc:1467] Processing CreateTablet for tablet 9574e973db55409f94ffe7e94dafca1f (DEFAULT_TABLE table=default.Table_468c64853fef4ebfa521100c23cb2620 [id=2b0e4f8550514fe9a0f06ff38ca4b973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.434664 745 tablet_service.cc:1467] Processing CreateTablet for tablet 266521a877ba4fd2bc4914e500f65eae (DEFAULT_TABLE table=default.Table_683d27a9444245be9ff47c13b148e508 [id=ddc535427f6f46248ab31cf5194f6b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.434808 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9574e973db55409f94ffe7e94dafca1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.434985 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266521a877ba4fd2bc4914e500f65eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.435217 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9cae71ae09d9491286d2d39a13dda36f (DEFAULT_TABLE table=default.Table_80b26a41f06c4b69a4cf5ef46f85fc8a [id=3eb6d42b9f914141b34592a22d1a1138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.435483 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cae71ae09d9491286d2d39a13dda36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.440521 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet dff155a5c2b0479a90e74993de6f402d
I20250115 14:52:05.440919 454 catalog_manager.cc:6596] ReplaceTablet: tablet dff155a5c2b0479a90e74993de6f402d deleted and replaced by tablet 026d05f8fb584da39000d83546b09e63
I20250115 14:52:05.441802 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1ff8a32badfe4ea8997fac383c0e5559" } new_table_name: "default.Table_eb5fca7507634f2d97b090af382eb6c9" modify_external_catalogs: true
I20250115 14:52:05.444868 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet a600fa0f0ec049dca2803bf7d2c74a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.445099 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 36514c5399924e98b3bc38004a33b919 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:05.445148 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e9febafb7e354e289cd3f864311f14cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9febafb7e354e289cd3f864311f14cb already in progress: creating tablet
I20250115 14:52:05.445425 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 9aefb12a03784eb895a17fd8c571aa24 (DEFAULT_TABLE table=default.Table_fa4c201ba84d4cf5902269c4c6e4f50c [id=bb1ad05767ba455e8d3c971c6557b498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.445652 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet e9febafb7e354e289cd3f864311f14cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.445957 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 266521a877ba4fd2bc4914e500f65eae (DEFAULT_TABLE table=default.Table_683d27a9444245be9ff47c13b148e508 [id=ddc535427f6f46248ab31cf5194f6b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.445986 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 9cae71ae09d9491286d2d39a13dda36f (DEFAULT_TABLE table=default.Table_80b26a41f06c4b69a4cf5ef46f85fc8a [id=3eb6d42b9f914141b34592a22d1a1138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.446018 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet dff155a5c2b0479a90e74993de6f402d with delete_type TABLET_DATA_DELETED (replaced by tablet 026d05f8fb584da39000d83546b09e63) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.446153 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 225d3c8c4c8c4f398ea99c4c32a67b57 (DEFAULT_TABLE table=default.Table_e215d4b1143c4ea49dde3acc698b7e60 [id=692f1b22ecbe4dc880e7757db8e7e02e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.446297 1013 tablet_service.cc:1467] Processing CreateTablet for tablet abccc613e1754839a4c7a8f5c448a852 (DEFAULT_TABLE table=default.Table_69ed5f68465549b38e7e27bf0d75eb2e [id=f08c5a8508224d27a45438fe8bd9390e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.445958 1015 tablet_service.cc:1467] Processing CreateTablet for tablet f5c1f0e0b6ee4448b9ba2f78d0117197 (DEFAULT_TABLE table=default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.446627 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 9574e973db55409f94ffe7e94dafca1f (DEFAULT_TABLE table=default.Table_468c64853fef4ebfa521100c23cb2620 [id=2b0e4f8550514fe9a0f06ff38ca4b973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.447780 879 tablet_service.cc:1467] Processing CreateTablet for tablet f5c1f0e0b6ee4448b9ba2f78d0117197 (DEFAULT_TABLE table=default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.448024 876 tablet_service.cc:1467] Processing CreateTablet for tablet 9574e973db55409f94ffe7e94dafca1f (DEFAULT_TABLE table=default.Table_468c64853fef4ebfa521100c23cb2620 [id=2b0e4f8550514fe9a0f06ff38ca4b973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.448150 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c1f0e0b6ee4448b9ba2f78d0117197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.448530 891 tablet_service.cc:1467] Processing CreateTablet for tablet abccc613e1754839a4c7a8f5c448a852 (DEFAULT_TABLE table=default.Table_69ed5f68465549b38e7e27bf0d75eb2e [id=f08c5a8508224d27a45438fe8bd9390e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.448601 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9574e973db55409f94ffe7e94dafca1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.449376 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abccc613e1754839a4c7a8f5c448a852. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.450608 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9cae71ae09d9491286d2d39a13dda36f (DEFAULT_TABLE table=default.Table_80b26a41f06c4b69a4cf5ef46f85fc8a [id=3eb6d42b9f914141b34592a22d1a1138]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.450982 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cae71ae09d9491286d2d39a13dda36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.456010 885 tablet_service.cc:1514] Processing DeleteTablet for tablet a600fa0f0ec049dca2803bf7d2c74a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.456394 875 tablet_service.cc:1514] Processing DeleteTablet for tablet dff155a5c2b0479a90e74993de6f402d with delete_type TABLET_DATA_DELETED (replaced by tablet 026d05f8fb584da39000d83546b09e63) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.456478 880 tablet_service.cc:1467] Processing CreateTablet for tablet 266521a877ba4fd2bc4914e500f65eae (DEFAULT_TABLE table=default.Table_683d27a9444245be9ff47c13b148e508 [id=ddc535427f6f46248ab31cf5194f6b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.456750 883 tablet_service.cc:1514] Processing DeleteTablet for tablet e9febafb7e354e289cd3f864311f14cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.456763 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266521a877ba4fd2bc4914e500f65eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.457005 886 tablet_service.cc:1467] Processing CreateTablet for tablet 225d3c8c4c8c4f398ea99c4c32a67b57 (DEFAULT_TABLE table=default.Table_e215d4b1143c4ea49dde3acc698b7e60 [id=692f1b22ecbe4dc880e7757db8e7e02e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.457269 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225d3c8c4c8c4f398ea99c4c32a67b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.458246 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:05.458997 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625]). Expected version 2 got 0
I20250115 14:52:05.462137 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6ce498ffd3f04c5d8eb642739a838903
W20250115 14:52:05.462860 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f82d944a7e04ace8e4addb3897d2c8e consensus state has no leader
I20250115 14:52:05.463594 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 36514c5399924e98b3bc38004a33b919 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.468160 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce498ffd3f04c5d8eb642739a838903 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.468456 1168 ts_tablet_manager.cc:1428] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20250115 14:52:05.468737 1168 tablet_bootstrap.cc:492] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:05.469224 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce498ffd3f04c5d8eb642739a838903 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.469534 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce498ffd3f04c5d8eb642739a838903 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.470103 749 tablet_service.cc:1514] Processing DeleteTablet for tablet dff155a5c2b0479a90e74993de6f402d with delete_type TABLET_DATA_DELETED (replaced by tablet 026d05f8fb584da39000d83546b09e63) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.470538 1166 tablet_replica.cc:331] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:05.470726 1166 raft_consensus.cc:2238] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.470878 1166 raft_consensus.cc:2267] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:05.471175 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b68c168048649c5b91600eb4302b0ce consensus state has no leader
I20250115 14:52:05.471385 1166 ts_tablet_manager.cc:1905] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.472419 1164 tablet_bootstrap.cc:492] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:05.472601 1162 tablet_bootstrap.cc:492] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:05.472609 1164 ts_tablet_manager.cc:1397] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.147s user 0.004s sys 0.001s
I20250115 14:52:05.472726 1162 ts_tablet_manager.cc:1397] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.166s user 0.003s sys 0.001s
I20250115 14:52:05.472828 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b85d63974ab418eb1921dd605fbf2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.473371 1162 raft_consensus.cc:357] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.473371 1164 raft_consensus.cc:357] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.473549 1162 raft_consensus.cc:383] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.473549 1164 raft_consensus.cc:383] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.473603 1162 raft_consensus.cc:738] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.473603 1164 raft_consensus.cc:738] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.473837 1164 consensus_queue.cc:260] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.479645 1164 ts_tablet_manager.cc:1428] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250115 14:52:05.482384 772 raft_consensus.cc:3054] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.482909 1164 tablet_bootstrap.cc:492] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.485221 1168 tablet_bootstrap.cc:654] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.485886 2013 consensus_queue.cc:1035] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:05.487522 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b84df010f86642d4b3585c0e774575bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.488104 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b84df010f86642d4b3585c0e774575bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.488698 882 tablet_service.cc:1467] Processing CreateTablet for tablet eb78eb2c5b9b44fba1a4e140a8e102d7 (DEFAULT_TABLE table=default.Table_57cca7a5af0b434b92b6e5c6eb5413f9 [id=cc7f2860ebc141cbae69eea0c8b524f1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:05.489851 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6ce498ffd3f04c5d8eb642739a838903 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ce498ffd3f04c5d8eb642739a838903 already in progress: creating tablet
I20250115 14:52:05.490808 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]). Expected version 1 got 0
I20250115 14:52:05.492580 749 tablet_service.cc:1467] Processing CreateTablet for tablet eb78eb2c5b9b44fba1a4e140a8e102d7 (DEFAULT_TABLE table=default.Table_57cca7a5af0b434b92b6e5c6eb5413f9 [id=cc7f2860ebc141cbae69eea0c8b524f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.492939 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb78eb2c5b9b44fba1a4e140a8e102d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.494068 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 338c0f6f39fd4335bbcda733b28b8bf8 (table default.Table_f367ac1a82704324a73f7079165a6e45 [id=6756cde43edd4d1e8a2768c101c4aa19]). Expected version 3 got 0
I20250115 14:52:05.494134 1166 ts_tablet_manager.cc:1918] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.494274 1166 log.cc:1198] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/dff155a5c2b0479a90e74993de6f402d
I20250115 14:52:05.494398 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet fbf94077e0e343b8938b081102dfd2af (table default.Table_bbd25ad975704fbcae339bf950fd1052 [id=00a89ba1b93f4132aae8fe6e748df77e]). Expected version 2 got 0
I20250115 14:52:05.494778 1166 ts_tablet_manager.cc:1939] T dff155a5c2b0479a90e74993de6f402d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:05.494685 453 catalog_manager.cc:5526] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:05.495187 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
I20250115 14:52:05.495466 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]). Expected version 1 got 0
I20250115 14:52:05.491590 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266521a877ba4fd2bc4914e500f65eae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.495934 1162 consensus_queue.cc:260] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.496454 1162 ts_tablet_manager.cc:1428] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20250115 14:52:05.499102 2013 raft_consensus.cc:491] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.499293 2013 raft_consensus.cc:513] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.499475 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dc4acc7152c24a22a0e7ac7de1943df3" } new_table_name: "default.Table_341db1cd78b3454789f331ecd640e902" modify_external_catalogs: true
I20250115 14:52:05.500191 1164 tablet_bootstrap.cc:654] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.500452 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ddfe9dfe14ba2b067f907a9d13538" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.500627 1047 raft_consensus.cc:2463] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:05.500993 847 leader_election.cc:304] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:05.501222 2138 raft_consensus.cc:2798] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:05.501284 2138 raft_consensus.cc:491] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:05.501335 2138 raft_consensus.cc:3054] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.501318 886 tablet_service.cc:1467] Processing CreateTablet for tablet cc9701156b0e48498666e34b3bec3043 (DEFAULT_TABLE table=default.Table_0218bb8722464e328f7582eacf088585 [id=c09306be04694782bf609e2e00fd773c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.501801 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb78eb2c5b9b44fba1a4e140a8e102d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.502293 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9701156b0e48498666e34b3bec3043. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.502286 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ddfe9dfe14ba2b067f907a9d13538" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:05.502700 848 leader_election.cc:343] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.503443 1024 tablet_service.cc:1467] Processing CreateTablet for tablet eb78eb2c5b9b44fba1a4e140a8e102d7 (DEFAULT_TABLE table=default.Table_57cca7a5af0b434b92b6e5c6eb5413f9 [id=cc7f2860ebc141cbae69eea0c8b524f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.503772 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb78eb2c5b9b44fba1a4e140a8e102d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.503881 2138 raft_consensus.cc:513] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.504413 2013 leader_election.cc:290] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.504997 2117 raft_consensus.cc:491] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.505115 2117 raft_consensus.cc:513] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.505345 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ddfe9dfe14ba2b067f907a9d13538" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:05.505511 1047 raft_consensus.cc:3054] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.505650 2117 leader_election.cc:290] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:05.508049 1047 raft_consensus.cc:2463] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:05.509876 847 leader_election.cc:304] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:05.510766 2138 leader_election.cc:290] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.510919 2138 raft_consensus.cc:2798] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:05.511008 2138 raft_consensus.cc:695] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:05.511224 2138 consensus_queue.cc:237] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.511518 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet dff155a5c2b0479a90e74993de6f402d (table default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]) successfully deleted
I20250115 14:52:05.512856 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf94077e0e343b8938b081102dfd2af" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:05.513247 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009ddfe9dfe14ba2b067f907a9d13538" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:05.513260 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf94077e0e343b8938b081102dfd2af" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.513443 1047 raft_consensus.cc:2388] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d7ecece2ab49e1be7079216517d13b in current term 1: Already voted for candidate 73c86158cc2641f182ea50a06634592d in this term.
W20250115 14:52:05.513622 848 leader_election.cc:343] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.515828 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59b36e986e146b9980c1f71599f32aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.516947 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f418504cc929429c936bfb16aa538464" } new_table_name: "default.Table_25bedec15a17479282c8ddf173d89ad8" modify_external_catalogs: true
I20250115 14:52:05.517321 1167 ts_tablet_manager.cc:1918] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.517362 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abccc613e1754839a4c7a8f5c448a852. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.517416 1167 log.cc:1198] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/496dec75500347699625a755dceed2bb
I20250115 14:52:05.517680 1167 ts_tablet_manager.cc:1939] T 496dec75500347699625a755dceed2bb P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:05.517930 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 613) took 2059 ms. Trace:
I20250115 14:52:05.518000 1167 rpcz_store.cc:276] 0115 14:52:03.458772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.459350 (+ 578us) service_pool.cc:225] Handling call
0115 14:52:05.356117 (+1896767us) ts_tablet_manager.cc:1145] Deleting tablet 496dec75500347699625a755dceed2bb
0115 14:52:05.356127 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.410372 (+ 54245us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.517234 (+106862us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.517917 (+ 683us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1896633}
I20250115 14:52:05.518340 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_14622d246cb34c0aa0c8527eebb4319b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:05.518438 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 496dec75500347699625a755dceed2bb (table default.Table_cb0308c238d7479dbcc041fe3d6e7eff [id=983e866df38543b0b2502165df44b94c]) successfully deleted
W20250115 14:52:05.518813 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14622d246cb34c0aa0c8527eebb4319b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.519888 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_679ebe5f3b29426082b31ce8a77da689"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.520320 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_679ebe5f3b29426082b31ce8a77da689 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.520486 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9574e973db55409f94ffe7e94dafca1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.524428 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cae71ae09d9491286d2d39a13dda36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.524658 710 leader_election.cc:304] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:05.524907 2117 raft_consensus.cc:3054] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.527300 2117 raft_consensus.cc:2743] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.528497 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9aefb12a03784eb895a17fd8c571aa24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.533387 1014 tablet_service.cc:1467] Processing CreateTablet for tablet cc9701156b0e48498666e34b3bec3043 (DEFAULT_TABLE table=default.Table_0218bb8722464e328f7582eacf088585 [id=c09306be04694782bf609e2e00fd773c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.533711 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9701156b0e48498666e34b3bec3043. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.533969 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c1f0e0b6ee4448b9ba2f78d0117197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.535755 1162 tablet_bootstrap.cc:492] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:05.536047 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 225d3c8c4c8c4f398ea99c4c32a67b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.536864 1167 tablet_replica.cc:331] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:05.537114 1167 raft_consensus.cc:2238] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:05.537392 1167 pending_rounds.cc:62] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:05.537508 1167 pending_rounds.cc:69] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558355843751936 op_type: NO_OP noop_request { }
I20250115 14:52:05.537678 1167 raft_consensus.cc:2883] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:05.537801 1167 raft_consensus.cc:2267] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.538460 1167 ts_tablet_manager.cc:1905] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.541282 1168 tablet_bootstrap.cc:492] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.541474 1168 ts_tablet_manager.cc:1397] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.000s
I20250115 14:52:05.542255 1168 raft_consensus.cc:357] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.542513 1168 raft_consensus.cc:383] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.542620 1168 raft_consensus.cc:738] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.542896 1168 consensus_queue.cc:260] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.543994 1168 ts_tablet_manager.cc:1428] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:05.544317 1168 tablet_bootstrap.cc:492] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:05.544212 739 tablet_service.cc:1467] Processing CreateTablet for tablet 288b0e455a854464940f65a2ab909d0b (DEFAULT_TABLE table=default.Table_61edbe7db3b44d02a60d60d865a86ec5 [id=5ae3104eeac943f69d0c1d79e148ab89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.545128 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 288b0e455a854464940f65a2ab909d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.545987 882 tablet_service.cc:1467] Processing CreateTablet for tablet 6afbdd0b585e4e7e82067dce061924d0 (DEFAULT_TABLE table=default.Table_eb0e928b8265482990bb5fbf1072985f [id=dfcc6db7be294db4a0f620ab93714f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.546164 1164 tablet_bootstrap.cc:492] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:05.546307 1164 ts_tablet_manager.cc:1397] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250115 14:52:05.546648 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6afbdd0b585e4e7e82067dce061924d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.547101 1164 raft_consensus.cc:357] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.547286 1164 raft_consensus.cc:383] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.547363 1164 raft_consensus.cc:738] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.547610 1164 consensus_queue.cc:260] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.548213 1164 ts_tablet_manager.cc:1428] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:05.548498 1164 tablet_bootstrap.cc:492] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.548669 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_55bce736d0234efa9eeee2a1745b955a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.549242 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55bce736d0234efa9eeee2a1745b955a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.549507 741 tablet_service.cc:1467] Processing CreateTablet for tablet abccc613e1754839a4c7a8f5c448a852 (DEFAULT_TABLE table=default.Table_69ed5f68465549b38e7e27bf0d75eb2e [id=f08c5a8508224d27a45438fe8bd9390e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.549870 1164 tablet_bootstrap.cc:654] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.550350 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a08bf8546b9148dc97c582984b4409c8" } modify_external_catalogs: true
I20250115 14:52:05.550534 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a08bf8546b9148dc97c582984b4409c8" } modify_external_catalogs: true
I20250115 14:52:05.550639 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abccc613e1754839a4c7a8f5c448a852. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.551407 893 tablet_service.cc:1467] Processing CreateTablet for tablet 288b0e455a854464940f65a2ab909d0b (DEFAULT_TABLE table=default.Table_61edbe7db3b44d02a60d60d865a86ec5 [id=5ae3104eeac943f69d0c1d79e148ab89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.551859 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 288b0e455a854464940f65a2ab909d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.555120 880 tablet_service.cc:1467] Processing CreateTablet for tablet 1de35551e26643b3b1e570ab8b2f6f56 (DEFAULT_TABLE table=default.Table_a7e733ef2d844d238685b7d25b53c151 [id=c40fb10d4df34d58a790b3d4e39b48e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.555462 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1de35551e26643b3b1e570ab8b2f6f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.557642 755 tablet_service.cc:1467] Processing CreateTablet for tablet 6afbdd0b585e4e7e82067dce061924d0 (DEFAULT_TABLE table=default.Table_eb0e928b8265482990bb5fbf1072985f [id=dfcc6db7be294db4a0f620ab93714f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.558001 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6afbdd0b585e4e7e82067dce061924d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.559059 2013 raft_consensus.cc:491] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.559165 2013 raft_consensus.cc:513] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.559760 2013 leader_election.cc:290] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.560632 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 288b0e455a854464940f65a2ab909d0b (DEFAULT_TABLE table=default.Table_61edbe7db3b44d02a60d60d865a86ec5 [id=5ae3104eeac943f69d0c1d79e148ab89]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.560999 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 288b0e455a854464940f65a2ab909d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.565140 886 tablet_service.cc:1467] Processing CreateTablet for tablet 567365040b6d4d6eba04112c8d4c3dfb (DEFAULT_TABLE table=default.Table_4ab29505c6634887aa99118862eccf5d [id=83320921ba86467b9ad31fde698cb2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.565541 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567365040b6d4d6eba04112c8d4c3dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.571491 889 tablet_service.cc:1467] Processing CreateTablet for tablet 1a676807ef38485ea49b0e92f5dc9862 (DEFAULT_TABLE table=default.Table_ae2b381e4eb84e6788f5d29ffc7a72f7 [id=a2079a8b5634463b83a77df10fd713a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.571815 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a676807ef38485ea49b0e92f5dc9862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.579515 2149 tablet.cc:1721] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:05.580498 2079 tablet.cc:1721] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:05.583356 749 tablet_service.cc:1467] Processing CreateTablet for tablet cc9701156b0e48498666e34b3bec3043 (DEFAULT_TABLE table=default.Table_0218bb8722464e328f7582eacf088585 [id=c09306be04694782bf609e2e00fd773c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.584257 751 tablet_service.cc:1467] Processing CreateTablet for tablet 1a676807ef38485ea49b0e92f5dc9862 (DEFAULT_TABLE table=default.Table_ae2b381e4eb84e6788f5d29ffc7a72f7 [id=a2079a8b5634463b83a77df10fd713a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.584250 745 tablet_service.cc:1467] Processing CreateTablet for tablet 567365040b6d4d6eba04112c8d4c3dfb (DEFAULT_TABLE table=default.Table_4ab29505c6634887aa99118862eccf5d [id=83320921ba86467b9ad31fde698cb2d1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:05.585028 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 84475.1us
I20250115 14:52:05.592749 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3963e79868284401a915cac5ef258155 (DEFAULT_TABLE table=default.Table_94471b2570804dbca5f0b06961b996e2 [id=bdf3eddde755447ea556a67bdc4de9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.593153 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3963e79868284401a915cac5ef258155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.595146 2013 raft_consensus.cc:491] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.595275 2013 raft_consensus.cc:513] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.596083 2013 leader_election.cc:290] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.596275 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e5d44b8064abbbf1b6419a1eca1ed" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:05.596760 848 leader_election.cc:343] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.590812 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.598691 1751 ts_tablet_manager.cc:1918] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.598917 1751 log.cc:1198] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d9ce65604c294021af93632ed073f627
I20250115 14:52:05.599169 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84bc24a71e431cb47b93f2b0dcaa90" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:05.599629 848 leader_election.cc:343] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.600370 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc9701156b0e48498666e34b3bec3043. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.601150 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a676807ef38485ea49b0e92f5dc9862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.607685 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567365040b6d4d6eba04112c8d4c3dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.609069 755 tablet_service.cc:1467] Processing CreateTablet for tablet 1de35551e26643b3b1e570ab8b2f6f56 (DEFAULT_TABLE table=default.Table_a7e733ef2d844d238685b7d25b53c151 [id=c40fb10d4df34d58a790b3d4e39b48e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.609485 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1de35551e26643b3b1e570ab8b2f6f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.612751 1876 raft_consensus.cc:491] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.612887 1876 raft_consensus.cc:513] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.613458 1876 leader_election.cc:290] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:05.613929 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759a4ebe7b0444128ddd31a78aad463e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:05.614298 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759a4ebe7b0444128ddd31a78aad463e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:05.614390 983 leader_election.cc:343] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.619243 891 tablet_service.cc:1467] Processing CreateTablet for tablet d443d39716d04b45b3151be80ed4320b (DEFAULT_TABLE table=default.Table_ab10a36d2dd14df0b5b7dde01478a485 [id=30c6c8d04ba34bc2ba1ef39fcdc5c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.619875 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d443d39716d04b45b3151be80ed4320b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.621577 848 consensus_peers.cc:487] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:05.625700 1167 ts_tablet_manager.cc:1918] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:05.625872 1167 log.cc:1198] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/caed629878594954842dfc6a40cc8b6d
I20250115 14:52:05.626256 1167 ts_tablet_manager.cc:1939] T caed629878594954842dfc6a40cc8b6d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:05.626549 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 617) took 2147 ms. Trace:
I20250115 14:52:05.626667 1167 rpcz_store.cc:276] 0115 14:52:03.478701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.481623 (+ 2922us) service_pool.cc:225] Handling call
0115 14:52:05.536813 (+2055190us) ts_tablet_manager.cc:1145] Deleting tablet caed629878594954842dfc6a40cc8b6d
0115 14:52:05.536823 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.542020 (+ 5197us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.625627 (+ 83607us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.626536 (+ 909us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2049593}
I20250115 14:52:05.627326 743 tablet_service.cc:1467] Processing CreateTablet for tablet f5c1f0e0b6ee4448b9ba2f78d0117197 (DEFAULT_TABLE table=default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.627878 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5c1f0e0b6ee4448b9ba2f78d0117197. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.632467 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 105b6148ae944e5c906e07d987177cc5 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.632997 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 35301651222249d0b3439e0638f599f2 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.633906 751 tablet_service.cc:1467] Processing CreateTablet for tablet 90c8e2aee66f440c8f3f2c4b661ad0af (DEFAULT_TABLE table=default.Table_3ae69bc8a5ee4225a1952f93b535c49b [id=9b4f2fa5beed4ebe92b88724f2cf1e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.634245 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c8e2aee66f440c8f3f2c4b661ad0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.636864 752 tablet_service.cc:1467] Processing CreateTablet for tablet d443d39716d04b45b3151be80ed4320b (DEFAULT_TABLE table=default.Table_ab10a36d2dd14df0b5b7dde01478a485 [id=30c6c8d04ba34bc2ba1ef39fcdc5c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.637183 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d443d39716d04b45b3151be80ed4320b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.639084 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6db88245ecc54e498a261791c7cc2103" } modify_external_catalogs: true
I20250115 14:52:05.639257 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6db88245ecc54e498a261791c7cc2103" } modify_external_catalogs: true
I20250115 14:52:05.627424 1168 tablet_bootstrap.cc:654] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.640735 455 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet fbf94077e0e343b8938b081102dfd2af (table default.Table_bbd25ad975704fbcae339bf950fd1052 [id=00a89ba1b93f4132aae8fe6e748df77e]). Expected version 2 got 0
I20250115 14:52:05.640918 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet caed629878594954842dfc6a40cc8b6d (table default.Table_544d606cf9af4accb670d91d7d717948 [id=b6aeb933c5e64e88a6d794a98650acbe]) successfully deleted
I20250115 14:52:05.641330 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 619) took 2118 ms. Trace:
I20250115 14:52:05.628377 889 tablet_service.cc:1467] Processing CreateTablet for tablet dcf442cc39c64f7ba3f78c083cd17b94 (DEFAULT_TABLE table=default.Table_73d45488e7de41b8ba166c493c6ce4d4 [id=c8e84fbcc59e4d32991c71a8e88a919b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.641407 1167 rpcz_store.cc:276] 0115 14:52:03.522882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.522933 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:05.641190 (+2118257us) ts_tablet_manager.cc:1145] Deleting tablet 02ce16fea4f9469c9fc06fa24f6554be
0115 14:52:05.641199 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.641306 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2118079}
I20250115 14:52:05.641419 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_340265c995a649808c582e4c4e6d0b22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:05.641671 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf442cc39c64f7ba3f78c083cd17b94. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.641827 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 02ce16fea4f9469c9fc06fa24f6554be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02ce16fea4f9469c9fc06fa24f6554be already in progress: creating tablet
W20250115 14:52:05.641918 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_340265c995a649808c582e4c4e6d0b22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.642103 1167 tablet_replica.cc:331] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:05.642365 1167 raft_consensus.cc:2238] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.642491 1167 raft_consensus.cc:2267] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.642709 752 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee6906a03f546bbaaecea44cf0e7a68 (DEFAULT_TABLE table=default.Table_05f540a8c52d40b1b3240deb036757ad [id=c43c43f4d17f4e1ba7d6dacf6cd1dbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.642927 1167 ts_tablet_manager.cc:1905] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.643023 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee6906a03f546bbaaecea44cf0e7a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.628654 882 tablet_service.cc:1467] Processing CreateTablet for tablet 90c8e2aee66f440c8f3f2c4b661ad0af (DEFAULT_TABLE table=default.Table_3ae69bc8a5ee4225a1952f93b535c49b [id=9b4f2fa5beed4ebe92b88724f2cf1e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.631480 886 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee6906a03f546bbaaecea44cf0e7a68 (DEFAULT_TABLE table=default.Table_05f540a8c52d40b1b3240deb036757ad [id=c43c43f4d17f4e1ba7d6dacf6cd1dbd0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:05.650791 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ff71ded3b074ff0979e2090131f4eaa consensus state has no leader
I20250115 14:52:05.651278 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_050541babd7b40668d93e38f18bef277"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.651784 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_050541babd7b40668d93e38f18bef277 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:05.653036 984 leader_election.cc:343] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.653126 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8be6e58a536c49c5add0072a13fa8be5" } modify_external_catalogs: true
I20250115 14:52:05.653173 984 leader_election.cc:304] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.653273 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8be6e58a536c49c5add0072a13fa8be5" } modify_external_catalogs: true
I20250115 14:52:05.653411 1876 raft_consensus.cc:2743] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.656523 745 tablet_service.cc:1467] Processing CreateTablet for tablet 3963e79868284401a915cac5ef258155 (DEFAULT_TABLE table=default.Table_94471b2570804dbca5f0b06961b996e2 [id=bdf3eddde755447ea556a67bdc4de9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.656899 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3963e79868284401a915cac5ef258155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.658068 2131 tablet.cc:1721] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:05.660504 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c8e2aee66f440c8f3f2c4b661ad0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.669574 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee6906a03f546bbaaecea44cf0e7a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.671787 1751 ts_tablet_manager.cc:1939] T d9ce65604c294021af93632ed073f627 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:05.673096 1164 tablet_bootstrap.cc:492] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:05.673288 1164 ts_tablet_manager.cc:1397] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.125s user 0.004s sys 0.000s
I20250115 14:52:05.673451 772 raft_consensus.cc:3054] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.674075 1164 raft_consensus.cc:357] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.674172 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 1de35551e26643b3b1e570ab8b2f6f56 (DEFAULT_TABLE table=default.Table_a7e733ef2d844d238685b7d25b53c151 [id=c40fb10d4df34d58a790b3d4e39b48e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.674279 1164 raft_consensus.cc:383] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.674345 1164 raft_consensus.cc:738] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.674851 1164 consensus_queue.cc:260] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.675508 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 1a676807ef38485ea49b0e92f5dc9862 (DEFAULT_TABLE table=default.Table_ae2b381e4eb84e6788f5d29ffc7a72f7 [id=a2079a8b5634463b83a77df10fd713a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.676563 1168 tablet_bootstrap.cc:492] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.676920 1164 ts_tablet_manager.cc:1428] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:52:05.677174 1164 tablet_bootstrap.cc:492] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.677335 1168 ts_tablet_manager.cc:1397] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.133s user 0.004s sys 0.000s
I20250115 14:52:05.677788 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d443d39716d04b45b3151be80ed4320b (DEFAULT_TABLE table=default.Table_ab10a36d2dd14df0b5b7dde01478a485 [id=30c6c8d04ba34bc2ba1ef39fcdc5c40d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.678148 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee6906a03f546bbaaecea44cf0e7a68 (DEFAULT_TABLE table=default.Table_05f540a8c52d40b1b3240deb036757ad [id=c43c43f4d17f4e1ba7d6dacf6cd1dbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.678510 1029 tablet_service.cc:1467] Processing CreateTablet for tablet dcf442cc39c64f7ba3f78c083cd17b94 (DEFAULT_TABLE table=default.Table_73d45488e7de41b8ba166c493c6ce4d4 [id=c8e84fbcc59e4d32991c71a8e88a919b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.679006 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:05.675117 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 567365040b6d4d6eba04112c8d4c3dfb (DEFAULT_TABLE table=default.Table_4ab29505c6634887aa99118862eccf5d [id=83320921ba86467b9ad31fde698cb2d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.680022 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 3963e79868284401a915cac5ef258155 (DEFAULT_TABLE table=default.Table_94471b2570804dbca5f0b06961b996e2 [id=bdf3eddde755447ea556a67bdc4de9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.680393 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 90c8e2aee66f440c8f3f2c4b661ad0af (DEFAULT_TABLE table=default.Table_3ae69bc8a5ee4225a1952f93b535c49b [id=9b4f2fa5beed4ebe92b88724f2cf1e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.681743 1164 tablet_bootstrap.cc:654] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.684648 1162 tablet_bootstrap.cc:654] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.686579 1168 raft_consensus.cc:357] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.686748 1168 raft_consensus.cc:383] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.686795 1168 raft_consensus.cc:738] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.688814 1167 ts_tablet_manager.cc:1918] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.693323 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84bc24a71e431cb47b93f2b0dcaa90" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.693498 1047 raft_consensus.cc:2463] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:05.693912 1046 raft_consensus.cc:1270] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:05.694248 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e5d44b8064abbbf1b6419a1eca1ed" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.694399 1049 raft_consensus.cc:2463] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:05.694447 1167 log.cc:1198] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/be63bae96c834a8bbbb1174c027935a2
I20250115 14:52:05.694734 1167 ts_tablet_manager.cc:1939] T be63bae96c834a8bbbb1174c027935a2 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:05.695078 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 624) took 2106 ms. Trace:
I20250115 14:52:05.695214 1167 rpcz_store.cc:276] 0115 14:52:03.588933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.589090 (+ 157us) service_pool.cc:225] Handling call
0115 14:52:05.642063 (+2052973us) ts_tablet_manager.cc:1145] Deleting tablet be63bae96c834a8bbbb1174c027935a2
0115 14:52:05.642072 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.645534 (+ 3462us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.688698 (+ 43164us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.695064 (+ 6366us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2052812}
I20250115 14:52:05.689045 1168 consensus_queue.cc:260] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.695973 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet be63bae96c834a8bbbb1174c027935a2 (table default.Table_705745b616194094bb03915f2f651122 [id=b2c6ee1ec113494cac7d1bf11a928714]) successfully deleted
I20250115 14:52:05.696349 1168 ts_tablet_manager.cc:1428] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250115 14:52:05.696794 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567365040b6d4d6eba04112c8d4c3dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.699554 847 leader_election.cc:304] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.699920 2138 raft_consensus.cc:2798] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:05.699999 2138 raft_consensus.cc:491] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:05.700053 2138 raft_consensus.cc:3054] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.702355 2138 raft_consensus.cc:513] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.702908 2138 leader_election.cc:290] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.704730 2138 consensus_queue.cc:1035] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:05.705622 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84bc24a71e431cb47b93f2b0dcaa90" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:05.705741 847 leader_election.cc:304] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.705798 1049 raft_consensus.cc:3054] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.706092 2138 raft_consensus.cc:2798] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:05.706151 2138 raft_consensus.cc:491] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:05.706200 2138 raft_consensus.cc:3054] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.708199 1049 raft_consensus.cc:2463] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:05.708259 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84bc24a71e431cb47b93f2b0dcaa90" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:05.709070 2138 raft_consensus.cc:513] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.709719 2138 leader_election.cc:290] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.710014 847 leader_election.cc:304] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:05.710263 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e5d44b8064abbbf1b6419a1eca1ed" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:05.710393 1049 raft_consensus.cc:3054] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.711483 2138 raft_consensus.cc:2798] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:05.711701 2138 raft_consensus.cc:695] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:05.711956 2138 consensus_queue.cc:237] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:05.712575 848 leader_election.cc:343] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.712921 1168 tablet_bootstrap.cc:492] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:05.713737 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e5d44b8064abbbf1b6419a1eca1ed" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:05.713912 1049 raft_consensus.cc:2463] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:05.714120 1162 tablet_bootstrap.cc:492] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:05.714238 1162 ts_tablet_manager.cc:1397] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.179s user 0.001s sys 0.007s
I20250115 14:52:05.713356 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 625) took 2050 ms. Trace:
I20250115 14:52:05.714778 1167 rpcz_store.cc:276] 0115 14:52:03.663173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.676889 (+ 13716us) service_pool.cc:225] Handling call
0115 14:52:05.696803 (+2019914us) ts_tablet_manager.cc:1145] Deleting tablet d9ce65604c294021af93632ed073f627
0115 14:52:05.713239 (+ 16436us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.713339 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2019739}
I20250115 14:52:05.714955 1162 raft_consensus.cc:357] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.715126 1162 raft_consensus.cc:383] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.715167 1162 raft_consensus.cc:738] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.715190 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 630) took 2001 ms. Trace:
I20250115 14:52:05.715255 1167 rpcz_store.cc:276] 0115 14:52:03.713463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.754976 (+ 41513us) service_pool.cc:225] Handling call
0115 14:52:05.715085 (+1960109us) ts_tablet_manager.cc:1145] Deleting tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb
0115 14:52:05.715095 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.715176 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1959944}
I20250115 14:52:05.715328 1162 consensus_queue.cc:260] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:05.714203 848 leader_election.cc:343] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.715801 1162 ts_tablet_manager.cc:1428] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:05.716024 1162 tablet_bootstrap.cc:492] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:05.716051 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 639) took 1800 ms. Trace:
W20250115 14:52:05.716102 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb already in progress: creating tablet
I20250115 14:52:05.716198 1167 rpcz_store.cc:276] 0115 14:52:03.915621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.915674 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:05.715428 (+1799754us) ts_tablet_manager.cc:1145] Deleting tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa
0115 14:52:05.715436 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.716035 (+ 599us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1793220}
W20250115 14:52:05.716598 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05bfcbb5c8c84cc1adf2f8c52d2d2daa already in progress: creating tablet
I20250115 14:52:05.716848 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 638) took 1818 ms. Trace:
I20250115 14:52:05.716954 1167 rpcz_store.cc:276] 0115 14:52:03.898063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.898115 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:05.716748 (+1818633us) ts_tablet_manager.cc:1145] Deleting tablet 9853dcdfee5f43de84798cfa0bef674e
0115 14:52:05.716759 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.716831 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794249}
I20250115 14:52:05.717419 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 640) took 1777 ms. Trace:
W20250115 14:52:05.717406 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9853dcdfee5f43de84798cfa0bef674e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9853dcdfee5f43de84798cfa0bef674e already in progress: creating tablet
I20250115 14:52:05.717518 1162 tablet_bootstrap.cc:654] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.717835 1167 rpcz_store.cc:276] 0115 14:52:03.939525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.939576 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:05.717333 (+1777757us) ts_tablet_manager.cc:1145] Deleting tablet 342bf8d4dc4244f982b1b63aaae0fb45
0115 14:52:05.717342 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.717407 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1777581}
I20250115 14:52:05.717677 847 leader_election.cc:304] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.718315 2013 raft_consensus.cc:2798] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:05.718418 2013 raft_consensus.cc:695] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:05.718612 2013 consensus_queue.cc:237] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.719254 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d9ce65604c294021af93632ed073f627 (table default.Table_79fecf866fd6405a9b7df222877fc2b0 [id=0748f5a86b034a96b1dfe1cd723d2438]) successfully deleted
I20250115 14:52:05.719586 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1de35551e26643b3b1e570ab8b2f6f56. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.724395 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet fa57c1fa96594a40b2551ad503127e4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa57c1fa96594a40b2551ad503127e4f already in progress: creating tablet
I20250115 14:52:05.724794 1751 tablet_replica.cc:331] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:05.725008 1751 raft_consensus.cc:2238] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.725286 1751 raft_consensus.cc:2267] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.726050 1751 ts_tablet_manager.cc:1905] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:05.728765 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 342bf8d4dc4244f982b1b63aaae0fb45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 342bf8d4dc4244f982b1b63aaae0fb45 already in progress: creating tablet
I20250115 14:52:05.728904 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 641) took 1720 ms. Trace:
I20250115 14:52:05.729305 1164 tablet_bootstrap.cc:492] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:05.729450 1164 ts_tablet_manager.cc:1397] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.001s
I20250115 14:52:05.729348 1167 rpcz_store.cc:276] 0115 14:52:04.008941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.009060 (+ 119us) service_pool.cc:225] Handling call
0115 14:52:05.728796 (+1719736us) ts_tablet_manager.cc:1145] Deleting tablet 655489ad9bd34be0a8283a295f583092
0115 14:52:05.728806 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.728886 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1719567}
W20250115 14:52:05.730031 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 655489ad9bd34be0a8283a295f583092 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 655489ad9bd34be0a8283a295f583092 already in progress: creating tablet
I20250115 14:52:05.730080 1167 tablet_replica.cc:331] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:05.730255 1167 raft_consensus.cc:2238] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.730180 1164 raft_consensus.cc:357] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.730383 1164 raft_consensus.cc:383] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.730449 1164 raft_consensus.cc:738] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.730901 1167 raft_consensus.cc:2267] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.731781 1167 ts_tablet_manager.cc:1905] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.732024 1164 consensus_queue.cc:260] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.732247 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d443d39716d04b45b3151be80ed4320b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.736346 1164 ts_tablet_manager.cc:1428] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:52:05.736806 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ee6906a03f546bbaaecea44cf0e7a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.740964 1164 tablet_bootstrap.cc:492] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.743306 1168 tablet_bootstrap.cc:654] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.745185 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a676807ef38485ea49b0e92f5dc9862. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.749555 1164 tablet_bootstrap.cc:654] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.749080 1162 tablet_bootstrap.cc:492] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:05.753288 1162 ts_tablet_manager.cc:1397] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20250115 14:52:05.754032 1162 raft_consensus.cc:357] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.754212 1162 raft_consensus.cc:383] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.754261 1162 raft_consensus.cc:738] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.754520 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3963e79868284401a915cac5ef258155. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.754423 1162 consensus_queue.cc:260] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.754979 1162 ts_tablet_manager.cc:1428] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:05.755235 1162 tablet_bootstrap.cc:492] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:05.756961 1162 tablet_bootstrap.cc:654] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.760584 2083 raft_consensus.cc:491] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.760664 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf442cc39c64f7ba3f78c083cd17b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.760728 2083 raft_consensus.cc:513] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.761336 2083 leader_election.cc:290] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.761739 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e94ea76507f4646909ac0c1b2c3136a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:05.762341 848 leader_election.cc:343] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.763476 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e94ea76507f4646909ac0c1b2c3136a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.763731 1049 raft_consensus.cc:2463] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:05.764252 847 leader_election.cc:304] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.764823 2083 raft_consensus.cc:2798] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:05.764890 2083 raft_consensus.cc:491] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:05.764942 2083 raft_consensus.cc:3054] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.766954 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c8e2aee66f440c8f3f2c4b661ad0af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.767210 2083 raft_consensus.cc:513] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.768594 2083 leader_election.cc:290] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.771950 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e94ea76507f4646909ac0c1b2c3136a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:05.772421 1049 raft_consensus.cc:3054] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.775077 1049 raft_consensus.cc:2463] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:05.775583 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e94ea76507f4646909ac0c1b2c3136a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:05.776068 848 leader_election.cc:343] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.778460 847 leader_election.cc:304] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.778848 2083 raft_consensus.cc:2798] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:05.778945 2083 raft_consensus.cc:695] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:05.779163 2083 consensus_queue.cc:237] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.791545 752 tablet_service.cc:1467] Processing CreateTablet for tablet dcf442cc39c64f7ba3f78c083cd17b94 (DEFAULT_TABLE table=default.Table_73d45488e7de41b8ba166c493c6ce4d4 [id=c8e84fbcc59e4d32991c71a8e88a919b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.791996 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf442cc39c64f7ba3f78c083cd17b94. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:05.794616 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edb5b4911c70478f8d02e476e9cb1769 consensus state has no leader
I20250115 14:52:05.794955 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 6afbdd0b585e4e7e82067dce061924d0 (DEFAULT_TABLE table=default.Table_eb0e928b8265482990bb5fbf1072985f [id=dfcc6db7be294db4a0f620ab93714f2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.795305 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6afbdd0b585e4e7e82067dce061924d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.797495 2200 raft_consensus.cc:491] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.797619 2200 raft_consensus.cc:513] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.798002 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4ced70cc9c3d4302b3856c7dc634d096" } modify_external_catalogs: true
I20250115 14:52:05.798166 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4ced70cc9c3d4302b3856c7dc634d096" } modify_external_catalogs: true
I20250115 14:52:05.798552 2200 leader_election.cc:290] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:05.799073 847 leader_election.cc:343] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 9d8c53fdf16243f0a614e48ec4ac94f4
I20250115 14:52:05.799654 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_78ff869a9d0c483a87b4d43f8bcac7c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.800169 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78ff869a9d0c483a87b4d43f8bcac7c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.801191 1162 tablet_bootstrap.cc:492] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:05.801299 1162 ts_tablet_manager.cc:1397] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.006s
I20250115 14:52:05.801347 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 171643c40c9b43ffb823f7c33789284d
I20250115 14:52:05.801935 1162 raft_consensus.cc:357] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.802088 1162 raft_consensus.cc:383] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.802002 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3b9fd4f4e09a40e99f015f099eef8306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:05.802126 1162 raft_consensus.cc:738] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.802292 1162 consensus_queue.cc:260] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:05.802408 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b9fd4f4e09a40e99f015f099eef8306 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.802706 1162 ts_tablet_manager.cc:1428] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250115 14:52:05.802946 1162 tablet_bootstrap.cc:492] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:05.804562 1162 tablet_bootstrap.cc:654] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.805914 1164 tablet_bootstrap.cc:492] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:05.806054 1164 ts_tablet_manager.cc:1397] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.003s
I20250115 14:52:05.806733 1164 raft_consensus.cc:357] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.806931 1164 raft_consensus.cc:383] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.807003 1164 raft_consensus.cc:738] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.807235 1164 consensus_queue.cc:260] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.807971 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 171643c40c9b43ffb823f7c33789284d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.808843 1168 tablet_bootstrap.cc:492] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.809100 1168 ts_tablet_manager.cc:1397] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.096s user 0.004s sys 0.001s
I20250115 14:52:05.811378 1164 ts_tablet_manager.cc:1428] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:05.819396 1164 tablet_bootstrap.cc:492] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.822813 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 171643c40c9b43ffb823f7c33789284d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.823115 1166 tablet_replica.cc:331] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:05.823338 1166 raft_consensus.cc:2238] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:05.823678 1166 raft_consensus.cc:2267] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.824618 1166 ts_tablet_manager.cc:1905] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.824851 1168 raft_consensus.cc:357] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.825161 1168 raft_consensus.cc:383] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.825240 1168 raft_consensus.cc:738] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.825528 1168 consensus_queue.cc:260] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.826359 1168 ts_tablet_manager.cc:1428] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250115 14:52:05.826768 1168 tablet_bootstrap.cc:492] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:05.748111 1167 ts_tablet_manager.cc:1918] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:05.827986 1167 log.cc:1198] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/4821224ac8f1472d8b1e8afc06a118a3
I20250115 14:52:05.827952 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d8c53fdf16243f0a614e48ec4ac94f4" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:05.832559 1166 ts_tablet_manager.cc:1918] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:05.833988 1166 log.cc:1198] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/171643c40c9b43ffb823f7c33789284d
I20250115 14:52:05.834350 1162 tablet_bootstrap.cc:492] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:05.834478 1162 ts_tablet_manager.cc:1397] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.000s
I20250115 14:52:05.832600 1164 tablet_bootstrap.cc:654] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.834875 1167 ts_tablet_manager.cc:1939] T 4821224ac8f1472d8b1e8afc06a118a3 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:05.835251 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 642) took 1785 ms. Trace:
I20250115 14:52:05.835381 1167 rpcz_store.cc:276] 0115 14:52:04.049914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.050555 (+ 641us) service_pool.cc:225] Handling call
0115 14:52:05.730040 (+1679485us) ts_tablet_manager.cc:1145] Deleting tablet 4821224ac8f1472d8b1e8afc06a118a3
0115 14:52:05.730049 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.745383 (+ 15334us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.748031 (+ 2648us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:05.835236 (+ 87205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679309}
I20250115 14:52:05.835844 1166 ts_tablet_manager.cc:1939] T 171643c40c9b43ffb823f7c33789284d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:05.835865 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 643) took 1758 ms. Trace:
I20250115 14:52:05.836009 1167 rpcz_store.cc:276] 0115 14:52:04.077033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.077293 (+ 260us) service_pool.cc:225] Handling call
0115 14:52:05.835747 (+1758454us) ts_tablet_manager.cc:1145] Deleting tablet 2e4c2707915e4e97ad6c66cb249c6d9e
0115 14:52:05.835757 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.835848 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1758274}
I20250115 14:52:05.836575 1167 tablet_replica.cc:331] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:05.836577 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 35301651222249d0b3439e0638f599f2
I20250115 14:52:05.836939 1167 raft_consensus.cc:2238] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.837044 446 catalog_manager.cc:6596] ReplaceTablet: tablet 35301651222249d0b3439e0638f599f2 deleted and replaced by tablet 7b6ea13ef1884d66bd1e7655df3e925d
I20250115 14:52:05.837235 1167 raft_consensus.cc:2267] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.837401 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ec25afcaa18b41b3906954d80092245d" } modify_external_catalogs: true
I20250115 14:52:05.837592 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ec25afcaa18b41b3906954d80092245d" } modify_external_catalogs: true
I20250115 14:52:05.838088 1167 ts_tablet_manager.cc:1905] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.839375 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5ef75e2dec224af78b8b5da8cf17abfe" } new_table_name: "default.Table_26ada8b381794c8e89848ce87729668d" modify_external_catalogs: true
I20250115 14:52:05.840550 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e59b36e986e146b9980c1f71599f32aa
I20250115 14:52:05.841167 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_edd5bd0ae3dd4bd994b4e746c1c14c9a" } new_table_name: "default.Table_19ac839d79bc40afae0076cf3c2c3877" modify_external_catalogs: true
I20250115 14:52:05.841212 2079 tablet.cc:1721] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:05.842518 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_21e00049c7364adb9a87e33cd7a04957"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:05.843003 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21e00049c7364adb9a87e33cd7a04957 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.847465 1164 tablet_bootstrap.cc:492] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:05.847605 1164 ts_tablet_manager.cc:1397] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20250115 14:52:05.847692 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet fbf5da59db79418498db09ec8ea3ac0a
I20250115 14:52:05.848361 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_38055c6c2fc748eb9fcefff19f2e92e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:05.848407 1164 raft_consensus.cc:357] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.848670 1164 raft_consensus.cc:383] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.848755 1164 raft_consensus.cc:738] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
W20250115 14:52:05.848894 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38055c6c2fc748eb9fcefff19f2e92e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.848994 1164 consensus_queue.cc:260] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.849052 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4d817129c1cf46a6bde83e97a293603b" } new_table_name: "default.Table_1b9d7f3106f14ea8866d92a6f857cf1d" modify_external_catalogs: true
I20250115 14:52:05.849484 1164 ts_tablet_manager.cc:1428] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:05.849750 1164 tablet_bootstrap.cc:492] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.850699 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 83097c5bfe364418a213236435edc40c from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.851274 1164 tablet_bootstrap.cc:654] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.854092 1751 ts_tablet_manager.cc:1918] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:05.854671 1751 log.cc:1198] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/759a4ebe7b0444128ddd31a78aad463e
I20250115 14:52:05.857424 1168 tablet_bootstrap.cc:654] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.859666 745 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf5da59db79418498db09ec8ea3ac0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.860375 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 171643c40c9b43ffb823f7c33789284d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.860419 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf5da59db79418498db09ec8ea3ac0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.860560 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_a08bf8546b9148dc97c582984b4409c8 [id=7628dbb6818f443680a97fef665d6d2b]) successfully deleted
I20250115 14:52:05.861017 2178 mvcc.cc:204] Tried to move back new op lower bound from 7114558364138483712 to 7114558363278516224. Current Snapshot: MvccSnapshot[applied={T|T < 7114558364138483712}]
W20250115 14:52:05.861081 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet fbf5da59db79418498db09ec8ea3ac0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbf5da59db79418498db09ec8ea3ac0a already in progress: creating tablet
I20250115 14:52:05.861222 2149 tablet.cc:1721] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:05.860917 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 35301651222249d0b3439e0638f599f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b6ea13ef1884d66bd1e7655df3e925d) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.861616 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 35301651222249d0b3439e0638f599f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b6ea13ef1884d66bd1e7655df3e925d) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.862049 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet e59b36e986e146b9980c1f71599f32aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.862047 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 35301651222249d0b3439e0638f599f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7b6ea13ef1884d66bd1e7655df3e925d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.862243 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e59b36e986e146b9980c1f71599f32aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.862689 886 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf5da59db79418498db09ec8ea3ac0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:05.863503 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2e4c2707915e4e97ad6c66cb249c6d9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e4c2707915e4e97ad6c66cb249c6d9e already in progress: creating tablet
W20250115 14:52:05.865062 848 leader_election.cc:343] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.865497 848 leader_election.cc:304] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.865865 2013 raft_consensus.cc:2743] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:05.867189 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 35301651222249d0b3439e0638f599f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35301651222249d0b3439e0638f599f2 already in progress: creating tablet
W20250115 14:52:05.872628 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1951b7ea6c0492b9920836515e7e5e5 consensus state has no leader
I20250115 14:52:05.872869 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 919ab6ec6f3745e082aa441dfebc235e from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:05.874085 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83cd48dfd1084d498153dfa11c0ab858 consensus state has no leader
I20250115 14:52:05.874316 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 72a97d102c174bc3a5e3a3c5762033b3 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.876982 1162 raft_consensus.cc:357] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.877509 1751 ts_tablet_manager.cc:1939] T 759a4ebe7b0444128ddd31a78aad463e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:05.877708 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71465ac2f92f4737803674a4249c1f85 consensus state has no leader
I20250115 14:52:05.877780 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4341977f363d40eeac71b7d33d1b43ac
I20250115 14:52:05.877601 1162 raft_consensus.cc:383] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.878062 1162 raft_consensus.cc:738] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.878700 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 4341977f363d40eeac71b7d33d1b43ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.878738 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 83097c5bfe364418a213236435edc40c
I20250115 14:52:05.879176 449 catalog_manager.cc:6596] ReplaceTablet: tablet 83097c5bfe364418a213236435edc40c deleted and replaced by tablet 4e39e13fb8c84ae18bac5e45d576a0ab
I20250115 14:52:05.881165 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 83097c5bfe364418a213236435edc40c with delete_type TABLET_DATA_DELETED (replaced by tablet 4e39e13fb8c84ae18bac5e45d576a0ab) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.885587 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e59b36e986e146b9980c1f71599f32aa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.892776 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 759a4ebe7b0444128ddd31a78aad463e (table default.Table_d1345e860a054b708e11f2f64347a3d7 [id=87e2283ba18a4525ab55a8316238aac7]) successfully deleted
I20250115 14:52:05.896365 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 83097c5bfe364418a213236435edc40c with delete_type TABLET_DATA_DELETED (replaced by tablet 4e39e13fb8c84ae18bac5e45d576a0ab) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.896435 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 919ab6ec6f3745e082aa441dfebc235e
I20250115 14:52:05.896904 454 catalog_manager.cc:6596] ReplaceTablet: tablet 919ab6ec6f3745e082aa441dfebc235e deleted and replaced by tablet 064f890830304758a6840a1502e59080
I20250115 14:52:05.897112 1166 tablet_replica.cc:331] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:05.897321 1166 raft_consensus.cc:2238] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.897524 1166 raft_consensus.cc:2267] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:05.897518 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e59b36e986e146b9980c1f71599f32aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e59b36e986e146b9980c1f71599f32aa already in progress: creating tablet
I20250115 14:52:05.898075 1166 ts_tablet_manager.cc:1905] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.898685 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 4341977f363d40eeac71b7d33d1b43ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.899364 2186 raft_consensus.cc:491] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.899845 1162 consensus_queue.cc:260] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:05.899495 2186 raft_consensus.cc:513] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.901016 2186 leader_election.cc:290] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:05.901357 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6cc08065494d9e85eeff5044b44ba6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:05.901654 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6cc08065494d9e85eeff5044b44ba6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:05.902159 984 leader_election.cc:343] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:05.902393 983 leader_election.cc:343] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.902469 983 leader_election.cc:304] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.902839 2186 raft_consensus.cc:2743] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.902184 1162 ts_tablet_manager.cc:1428] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.068s user 0.003s sys 0.001s
I20250115 14:52:05.903267 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 919ab6ec6f3745e082aa441dfebc235e with delete_type TABLET_DATA_DELETED (replaced by tablet 064f890830304758a6840a1502e59080) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.903512 1751 tablet_replica.cc:331] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:05.904278 1166 ts_tablet_manager.cc:1918] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:05.904466 1166 log.cc:1198] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/83097c5bfe364418a213236435edc40c
I20250115 14:52:05.904829 1166 ts_tablet_manager.cc:1939] T 83097c5bfe364418a213236435edc40c P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:05.905128 1164 tablet_bootstrap.cc:492] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:05.905280 1164 ts_tablet_manager.cc:1397] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.006s
I20250115 14:52:05.905975 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 83097c5bfe364418a213236435edc40c (table default.Table_81803594c5c741619ae708256092f827 [id=caa6adb7563c41088e5120993c46c54f]) successfully deleted
I20250115 14:52:05.906006 1164 raft_consensus.cc:357] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.906209 1164 raft_consensus.cc:383] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.906286 1164 raft_consensus.cc:738] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.906337 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 919ab6ec6f3745e082aa441dfebc235e with delete_type TABLET_DATA_DELETED (replaced by tablet 064f890830304758a6840a1502e59080) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.906497 1164 consensus_queue.cc:260] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:05.907631 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4341977f363d40eeac71b7d33d1b43ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4341977f363d40eeac71b7d33d1b43ac already in progress: creating tablet
W20250115 14:52:05.909169 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 919ab6ec6f3745e082aa441dfebc235e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 919ab6ec6f3745e082aa441dfebc235e already in progress: creating tablet
I20250115 14:52:05.911074 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 4341977f363d40eeac71b7d33d1b43ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.911824 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 83097c5bfe364418a213236435edc40c with delete_type TABLET_DATA_DELETED (replaced by tablet 4e39e13fb8c84ae18bac5e45d576a0ab) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.911928 1164 ts_tablet_manager.cc:1428] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:52:05.913648 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 919ab6ec6f3745e082aa441dfebc235e with delete_type TABLET_DATA_DELETED (replaced by tablet 064f890830304758a6840a1502e59080) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.915405 1164 tablet_bootstrap.cc:492] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:05.916254 2079 tablet.cc:1721] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:05.917994 2149 tablet.cc:1721] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:05.918396 1164 tablet_bootstrap.cc:654] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.924921 447 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet fbf94077e0e343b8938b081102dfd2af (table default.Table_bbd25ad975704fbcae339bf950fd1052 [id=00a89ba1b93f4132aae8fe6e748df77e]). Expected version 2 got 0
I20250115 14:52:05.925176 447 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]). Expected version 3 got 0
I20250115 14:52:05.927577 449 catalog_manager.cc:5526] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:05.928241 449 catalog_manager.cc:5526] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:05.928474 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 453adba8c772446c9e24f20eed56a951 (DEFAULT_TABLE table=default.Table_799a6df4f73a45c9a57ced3b6a102091 [id=3ed28186b56f489cb5b14c412b20cd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.929239 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d5dd41d42331494e9c1b021816d0287e
I20250115 14:52:05.930516 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet d5dd41d42331494e9c1b021816d0287e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:05.931595 745 tablet_service.cc:1514] Processing DeleteTablet for tablet d5dd41d42331494e9c1b021816d0287e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.931721 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 45f124134eab4babac145404c62e7d2a (DEFAULT_TABLE table=default.Table_40e515dbe043466f8d1ca690f14c6f42 [id=7efc6a9de6f346ffb91bad315163c48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.934098 1168 tablet_bootstrap.cc:492] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.934278 1168 ts_tablet_manager.cc:1397] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.000s
I20250115 14:52:05.935005 1168 raft_consensus.cc:357] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.935269 1168 raft_consensus.cc:383] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.935654 745 tablet_service.cc:1467] Processing CreateTablet for tablet ba21cf54bc074d95b456fd70a9286001 (DEFAULT_TABLE table=default.Table_d43d582508924d259b43a911c5d2c876 [id=d058d09355784e3689042f31e8014224]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.936129 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba21cf54bc074d95b456fd70a9286001. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.936091 1011 tablet_service.cc:1467] Processing CreateTablet for tablet ba21cf54bc074d95b456fd70a9286001 (DEFAULT_TABLE table=default.Table_d43d582508924d259b43a911c5d2c876 [id=d058d09355784e3689042f31e8014224]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.938001 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_526b537d21ae4762b6881e46346414c6 [id=a724f6e4ad16485b8ee2445bb072be3e] alter complete (version 2)
I20250115 14:52:05.940363 1168 raft_consensus.cc:738] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:05.940613 1168 consensus_queue.cc:260] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.941262 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bc975b044dd439dbd1f8f9a82cce751 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.941869 1168 ts_tablet_manager.cc:1428] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250115 14:52:05.942729 739 tablet_service.cc:1514] Processing DeleteTablet for tablet 009ddfe9dfe14ba2b067f907a9d13538 with delete_type TABLET_DATA_DELETED (replaced by tablet ee5671e8a08d47f7bea9fc6af2f53b2d) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:05.942670 1168 tablet_bootstrap.cc:492] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:05.944180 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 96e309ad749b4bafacc95a9614f0f903 (DEFAULT_TABLE table=default.Table_d42cd252db384a70b6ef1a2595274afa [id=ca0b7e13dd94426ca92f1910d2984624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.944671 1010 tablet_service.cc:1467] Processing CreateTablet for tablet e663111811b34a578bc344c59be8649e (DEFAULT_TABLE table=default.Table_9da42aab96b54c6f84d977c95677d7e3 [id=d9dac5a3219b48419840e02c2a26c1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.945425 1029 tablet_service.cc:1467] Processing CreateTablet for tablet e185963130bf49a1af88ab9e14e8e1bc (DEFAULT_TABLE table=default.Table_5ea7727ee96e426594a8c8af14d9545d [id=d4ecfa4ec6ea48ec8d24a1ccb03c8207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.946223 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 90df7f51fb044083920d8194823be35f (DEFAULT_TABLE table=default.Table_1c56d7edc9fa481db33c058487afde17 [id=ee990f5450fa4e80878557d421f10492]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.946987 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 7e600acd0ab74940a14f3022dbfa747f (DEFAULT_TABLE table=default.Table_74edc08b682e4c5cb696e54fcadaf5f1 [id=ef4314fd986f48e6b62461571b4dcae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.947795 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b54807a4e9644b86a89670f57ae154a3 (DEFAULT_TABLE table=default.Table_9339e2f34b994dd7a9f6385ee32f1b05 [id=a1658e0c367c496e88ea66c849bd2699]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.948629 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 2119fed44f55434f94698a2f9d45d7aa (DEFAULT_TABLE table=default.Table_84124eeb8c424548bd8b58245bb7823f [id=9ba1ec3a3a6f47dc832d149ac3edf4a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.949755 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 96369c35ca0d4a62b0676cac3dc4a813 (DEFAULT_TABLE table=default.Table_ab3fe6097891488eb31670fe723ef7d6 [id=03160ff24687408084cd57046d8b17b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.951335 741 tablet_service.cc:1467] Processing CreateTablet for tablet e663111811b34a578bc344c59be8649e (DEFAULT_TABLE table=default.Table_9da42aab96b54c6f84d977c95677d7e3 [id=d9dac5a3219b48419840e02c2a26c1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.951735 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e663111811b34a578bc344c59be8649e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.951782 748 tablet_service.cc:1467] Processing CreateTablet for tablet 96369c35ca0d4a62b0676cac3dc4a813 (DEFAULT_TABLE table=default.Table_ab3fe6097891488eb31670fe723ef7d6 [id=03160ff24687408084cd57046d8b17b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.951875 743 tablet_service.cc:1467] Processing CreateTablet for tablet 45f124134eab4babac145404c62e7d2a (DEFAULT_TABLE table=default.Table_40e515dbe043466f8d1ca690f14c6f42 [id=7efc6a9de6f346ffb91bad315163c48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.951987 755 tablet_service.cc:1467] Processing CreateTablet for tablet e185963130bf49a1af88ab9e14e8e1bc (DEFAULT_TABLE table=default.Table_5ea7727ee96e426594a8c8af14d9545d [id=d4ecfa4ec6ea48ec8d24a1ccb03c8207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.952145 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96369c35ca0d4a62b0676cac3dc4a813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.952355 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e185963130bf49a1af88ab9e14e8e1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.952600 756 tablet_service.cc:1467] Processing CreateTablet for tablet 2abbe321d0fc421d9a45d4dccc74e161 (DEFAULT_TABLE table=default.Table_b21a9aaeef5245b2813b04521d7a113b [id=c209d33d5cbf4e0a8cd0c371915ad399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.952739 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f124134eab4babac145404c62e7d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.952905 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abbe321d0fc421d9a45d4dccc74e161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.953246 752 tablet_service.cc:1467] Processing CreateTablet for tablet 453adba8c772446c9e24f20eed56a951 (DEFAULT_TABLE table=default.Table_799a6df4f73a45c9a57ced3b6a102091 [id=3ed28186b56f489cb5b14c412b20cd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.953279 750 tablet_service.cc:1467] Processing CreateTablet for tablet b54807a4e9644b86a89670f57ae154a3 (DEFAULT_TABLE table=default.Table_9339e2f34b994dd7a9f6385ee32f1b05 [id=a1658e0c367c496e88ea66c849bd2699]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.953585 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54807a4e9644b86a89670f57ae154a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.959132 447 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1]). Expected version 3 got 0
I20250115 14:52:05.962357 1168 tablet_bootstrap.cc:654] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:05.964684 449 catalog_manager.cc:5526] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:05.965744 2186 raft_consensus.cc:491] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.965875 2186 raft_consensus.cc:513] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.966468 2186 leader_election.cc:290] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:05.967044 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0d97fa57484b9d8cb879fedda61612" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:05.967418 2131 tablet.cc:1721] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250115 14:52:05.968161 984 leader_election.cc:343] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.969339 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0d97fa57484b9d8cb879fedda61612" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:05.969476 1049 raft_consensus.cc:1270] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250115 14:52:05.969779 983 leader_election.cc:343] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:05.969870 983 leader_election.cc:304] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:05.969971 2013 consensus_queue.cc:1035] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:05.970109 1168 tablet_bootstrap.cc:492] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:05.970228 1168 ts_tablet_manager.cc:1397] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.001s
I20250115 14:52:05.970309 2186 raft_consensus.cc:2743] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:05.970928 886 tablet_service.cc:1467] Processing CreateTablet for tablet 453adba8c772446c9e24f20eed56a951 (DEFAULT_TABLE table=default.Table_799a6df4f73a45c9a57ced3b6a102091 [id=3ed28186b56f489cb5b14c412b20cd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.971267 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453adba8c772446c9e24f20eed56a951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.971745 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:05.974673 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 2abbe321d0fc421d9a45d4dccc74e161 (DEFAULT_TABLE table=default.Table_b21a9aaeef5245b2813b04521d7a113b [id=c209d33d5cbf4e0a8cd0c371915ad399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.976380 1166 tablet_replica.cc:331] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:05.976423 739 tablet_service.cc:1467] Processing CreateTablet for tablet 96e309ad749b4bafacc95a9614f0f903 (DEFAULT_TABLE table=default.Table_d42cd252db384a70b6ef1a2595274afa [id=ca0b7e13dd94426ca92f1910d2984624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.976706 1166 raft_consensus.cc:2238] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:05.976840 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e309ad749b4bafacc95a9614f0f903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.976974 1166 raft_consensus.cc:2267] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:05.977896 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d5dd41d42331494e9c1b021816d0287e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:05.978369 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 0159e9b3b87844a393d958668e318124 (DEFAULT_TABLE table=default.Table_00fd62cc66984eecb416652888d31733 [id=c26cabdeb5a643759e2564c489a8f282]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.978690 2013 raft_consensus.cc:491] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:05.978719 886 tablet_service.cc:1467] Processing CreateTablet for tablet 45f124134eab4babac145404c62e7d2a (DEFAULT_TABLE table=default.Table_40e515dbe043466f8d1ca690f14c6f42 [id=7efc6a9de6f346ffb91bad315163c48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.978808 2013 raft_consensus.cc:513] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.979010 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f124134eab4babac145404c62e7d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.979334 2013 leader_election.cc:290] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.979815 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2ee5dfdef14ec092a2f9d4bfd295a6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:05.979977 1047 raft_consensus.cc:2463] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:05.980362 847 leader_election.cc:304] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:05.980793 2013 raft_consensus.cc:2798] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:05.980861 2013 raft_consensus.cc:491] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:05.980911 2013 raft_consensus.cc:3054] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.983249 2013 raft_consensus.cc:513] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.984166 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2ee5dfdef14ec092a2f9d4bfd295a6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:05.984341 1047 raft_consensus.cc:3054] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:05.984931 2013 leader_election.cc:290] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:05.985702 1166 ts_tablet_manager.cc:1905] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:05.986634 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7e600acd0ab74940a14f3022dbfa747f (DEFAULT_TABLE table=default.Table_74edc08b682e4c5cb696e54fcadaf5f1 [id=ef4314fd986f48e6b62461571b4dcae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.986634 749 tablet_service.cc:1467] Processing CreateTablet for tablet 90df7f51fb044083920d8194823be35f (DEFAULT_TABLE table=default.Table_1c56d7edc9fa481db33c058487afde17 [id=ee990f5450fa4e80878557d421f10492]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.987201 744 tablet_service.cc:1467] Processing CreateTablet for tablet c3f78abe3a6049cb90d45e5f9e61b061 (DEFAULT_TABLE table=default.Table_fc7747535f7b4449867ef0aeb96f88f4 [id=0bc02e2bdda844aaae08fd1c850c984d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.987519 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2119fed44f55434f94698a2f9d45d7aa (DEFAULT_TABLE table=default.Table_84124eeb8c424548bd8b58245bb7823f [id=9ba1ec3a3a6f47dc832d149ac3edf4a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.987766 740 tablet_service.cc:1467] Processing CreateTablet for tablet 0159e9b3b87844a393d958668e318124 (DEFAULT_TABLE table=default.Table_00fd62cc66984eecb416652888d31733 [id=c26cabdeb5a643759e2564c489a8f282]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:05.992637 1047 raft_consensus.cc:2463] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:05.993042 847 leader_election.cc:304] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:05.993353 2202 raft_consensus.cc:2798] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:05.993515 2202 raft_consensus.cc:695] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:05.993587 447 master_service.cc:946] ReplaceTablet: received request to replace tablet abccc613e1754839a4c7a8f5c448a852 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:05.993902 2202 consensus_queue.cc:237] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.994376 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90df7f51fb044083920d8194823be35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.994626 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2119fed44f55434f94698a2f9d45d7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:05.997776 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_201c8e612d7d4a85a6d4d4b0a5095254" } new_table_name: "default.Table_449c8d18596548be9715ed0c45d1bc4f" modify_external_catalogs: true
I20250115 14:52:05.999001 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_76b7b4a0c17041068bfb2be07cf21267"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:05.999209 1168 raft_consensus.cc:357] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:05.999382 1168 raft_consensus.cc:383] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:05.999428 1168 raft_consensus.cc:738] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
W20250115 14:52:05.999459 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76b7b4a0c17041068bfb2be07cf21267 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:05.999590 1168 consensus_queue.cc:260] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.000372 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_f367ac1a82704324a73f7079165a6e45 [id=6756cde43edd4d1e8a2768c101c4aa19] alter complete (version 3)
I20250115 14:52:06.001354 1168 ts_tablet_manager.cc:1428] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20250115 14:52:06.002367 1168 tablet_bootstrap.cc:492] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:06.004184 1168 tablet_bootstrap.cc:654] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.004236 1162 tablet_bootstrap.cc:492] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.006695 1162 tablet_bootstrap.cc:654] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.007623 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45f124134eab4babac145404c62e7d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.008193 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b82924e735846b2914520684d1b8fc7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:06.008930 882 tablet_service.cc:1467] Processing CreateTablet for tablet 96e309ad749b4bafacc95a9614f0f903 (DEFAULT_TABLE table=default.Table_d42cd252db384a70b6ef1a2595274afa [id=ca0b7e13dd94426ca92f1910d2984624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.009228 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e309ad749b4bafacc95a9614f0f903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.010644 1167 ts_tablet_manager.cc:1918] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.012100 889 tablet_service.cc:1467] Processing CreateTablet for tablet e663111811b34a578bc344c59be8649e (DEFAULT_TABLE table=default.Table_9da42aab96b54c6f84d977c95677d7e3 [id=d9dac5a3219b48419840e02c2a26c1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.012701 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba21cf54bc074d95b456fd70a9286001. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.012737 880 tablet_service.cc:1467] Processing CreateTablet for tablet 90df7f51fb044083920d8194823be35f (DEFAULT_TABLE table=default.Table_1c56d7edc9fa481db33c058487afde17 [id=ee990f5450fa4e80878557d421f10492]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.012230 1167 log.cc:1198] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d195be2d543f4a8693045a024912c43a
I20250115 14:52:06.011345 886 tablet_service.cc:1467] Processing CreateTablet for tablet ba21cf54bc074d95b456fd70a9286001 (DEFAULT_TABLE table=default.Table_d43d582508924d259b43a911c5d2c876 [id=d058d09355784e3689042f31e8014224]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.013123 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e309ad749b4bafacc95a9614f0f903. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.014777 883 tablet_service.cc:1467] Processing CreateTablet for tablet 7e600acd0ab74940a14f3022dbfa747f (DEFAULT_TABLE table=default.Table_74edc08b682e4c5cb696e54fcadaf5f1 [id=ef4314fd986f48e6b62461571b4dcae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.012399 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e663111811b34a578bc344c59be8649e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.012415 891 tablet_service.cc:1514] Processing DeleteTablet for tablet d71a1ecd900942e5ae788c9a28c70a7a with delete_type TABLET_DATA_DELETED (replaced by tablet 0651be54e0b244a4bace9f47efc9b58a) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:06.012566 893 tablet_service.cc:1467] Processing CreateTablet for tablet e185963130bf49a1af88ab9e14e8e1bc (DEFAULT_TABLE table=default.Table_5ea7727ee96e426594a8c8af14d9545d [id=d4ecfa4ec6ea48ec8d24a1ccb03c8207]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.016674 2186 raft_consensus.cc:491] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.016673 879 tablet_service.cc:1467] Processing CreateTablet for tablet b54807a4e9644b86a89670f57ae154a3 (DEFAULT_TABLE table=default.Table_9339e2f34b994dd7a9f6385ee32f1b05 [id=a1658e0c367c496e88ea66c849bd2699]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.016794 2186 raft_consensus.cc:513] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.017356 2186 leader_election.cc:290] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.016819 1167 ts_tablet_manager.cc:1939] T d195be2d543f4a8693045a024912c43a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:06.016688 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e663111811b34a578bc344c59be8649e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.018018 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e600acd0ab74940a14f3022dbfa747f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.018708 891 tablet_service.cc:1467] Processing CreateTablet for tablet 2119fed44f55434f94698a2f9d45d7aa (DEFAULT_TABLE table=default.Table_84124eeb8c424548bd8b58245bb7823f [id=9ba1ec3a3a6f47dc832d149ac3edf4a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.020390 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90df7f51fb044083920d8194823be35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.020503 875 tablet_service.cc:1467] Processing CreateTablet for tablet 96369c35ca0d4a62b0676cac3dc4a813 (DEFAULT_TABLE table=default.Table_ab3fe6097891488eb31670fe723ef7d6 [id=03160ff24687408084cd57046d8b17b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.020964 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96369c35ca0d4a62b0676cac3dc4a813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.023293 876 tablet_service.cc:1467] Processing CreateTablet for tablet 2abbe321d0fc421d9a45d4dccc74e161 (DEFAULT_TABLE table=default.Table_b21a9aaeef5245b2813b04521d7a113b [id=c209d33d5cbf4e0a8cd0c371915ad399]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.024389 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e185963130bf49a1af88ab9e14e8e1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.024856 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2119fed44f55434f94698a2f9d45d7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.026494 882 tablet_service.cc:1467] Processing CreateTablet for tablet 0159e9b3b87844a393d958668e318124 (DEFAULT_TABLE table=default.Table_00fd62cc66984eecb416652888d31733 [id=c26cabdeb5a643759e2564c489a8f282]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.016682 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba21cf54bc074d95b456fd70a9286001. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.028429 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54807a4e9644b86a89670f57ae154a3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:06.028954 847 consensus_peers.cc:487] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a915e0c0a30940ed8a6233f117e2a560. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:06.029222 885 tablet_service.cc:1467] Processing CreateTablet for tablet c3f78abe3a6049cb90d45e5f9e61b061 (DEFAULT_TABLE table=default.Table_fc7747535f7b4449867ef0aeb96f88f4 [id=0bc02e2bdda844aaae08fd1c850c984d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.031850 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abbe321d0fc421d9a45d4dccc74e161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.032387 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96369c35ca0d4a62b0676cac3dc4a813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.036381 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f78abe3a6049cb90d45e5f9e61b061. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.037930 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee8b63cc21145d8993b058d8c6d23c7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:06.038503 983 leader_election.cc:343] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.039105 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e600acd0ab74940a14f3022dbfa747f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.043502 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0159e9b3b87844a393d958668e318124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.045015 1162 tablet_bootstrap.cc:492] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:06.045141 1162 ts_tablet_manager.cc:1397] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.008s
I20250115 14:52:06.045811 1162 raft_consensus.cc:357] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.046046 1162 raft_consensus.cc:383] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.046142 1162 raft_consensus.cc:738] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.046341 2186 raft_consensus.cc:491] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.046402 1162 consensus_queue.cc:260] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.046465 2186 raft_consensus.cc:513] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.047008 2186 leader_election.cc:290] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.046994 1162 ts_tablet_manager.cc:1428] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:06.047369 1162 tablet_bootstrap.cc:492] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.048743 1162 tablet_bootstrap.cc:654] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.050330 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f78abe3a6049cb90d45e5f9e61b061. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.042047 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54807a4e9644b86a89670f57ae154a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.050534 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e600acd0ab74940a14f3022dbfa747f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.053383 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90df7f51fb044083920d8194823be35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.054343 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453adba8c772446c9e24f20eed56a951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.056310 1751 raft_consensus.cc:2238] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.056453 1751 raft_consensus.cc:2267] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.056512 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e185963130bf49a1af88ab9e14e8e1bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.058022 1751 ts_tablet_manager.cc:1905] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.059590 2138 raft_consensus.cc:491] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.059693 2138 raft_consensus.cc:513] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.060580 2138 leader_election.cc:290] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.060873 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2abbe321d0fc421d9a45d4dccc74e161. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.061125 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2119fed44f55434f94698a2f9d45d7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.040832 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 646) took 1914 ms. Trace:
I20250115 14:52:06.064476 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0159e9b3b87844a393d958668e318124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.068410 1167 rpcz_store.cc:276] 0115 14:52:04.126341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.126388 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:05.836503 (+1710115us) ts_tablet_manager.cc:1145] Deleting tablet d195be2d543f4a8693045a024912c43a
0115 14:52:05.836515 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:05.887668 (+ 51153us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.005762 (+118094us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.040430 (+ 34668us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1709925}
W20250115 14:52:06.068606 847 leader_election.cc:343] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 104282a09c024bbaafeb5f28f91dadcb
I20250115 14:52:06.068838 1162 tablet_bootstrap.cc:492] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:06.068815 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 647) took 1937 ms. Trace:
I20250115 14:52:06.068953 1162 ts_tablet_manager.cc:1397] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20250115 14:52:06.068886 1167 rpcz_store.cc:276] 0115 14:52:04.130958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.134066 (+ 3108us) service_pool.cc:225] Handling call
0115 14:52:06.068680 (+1934614us) ts_tablet_manager.cc:1145] Deleting tablet e180e37a1d5d46d78380323662b80f85
0115 14:52:06.068690 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.068800 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1934441}
I20250115 14:52:06.069141 1167 tablet_replica.cc:331] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:06.069314 1167 raft_consensus.cc:2238] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.069630 1162 raft_consensus.cc:357] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.069815 1162 raft_consensus.cc:383] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.069866 1162 raft_consensus.cc:738] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.049386 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b85d63974ab418eb1921dd605fbf2b0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:06.073117 1167 raft_consensus.cc:2267] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.073628 1167 ts_tablet_manager.cc:1905] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.073979 1751 ts_tablet_manager.cc:1918] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.079807 1751 log.cc:1198] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4abde784f9c948cb96e6cc93c30817bd
I20250115 14:52:06.080422 1751 ts_tablet_manager.cc:1939] T 4abde784f9c948cb96e6cc93c30817bd P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:06.074191 983 leader_election.cc:343] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.075809 1162 consensus_queue.cc:260] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.081727 1162 ts_tablet_manager.cc:1428] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250115 14:52:06.082005 1162 tablet_bootstrap.cc:492] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.082172 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 8af81a1841354f418cb81392cf3c252f (DEFAULT_TABLE table=default.Table_4c53ae523be9439e825de4b84817ff71 [id=66ca1d60df8040118e79a1e8ac24d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.082566 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af81a1841354f418cb81392cf3c252f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.083287 1162 tablet_bootstrap.cc:654] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.080997 1024 tablet_service.cc:1467] Processing CreateTablet for tablet c3f78abe3a6049cb90d45e5f9e61b061 (DEFAULT_TABLE table=default.Table_fc7747535f7b4449867ef0aeb96f88f4 [id=0bc02e2bdda844aaae08fd1c850c984d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.084101 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f78abe3a6049cb90d45e5f9e61b061. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.084885 893 tablet_service.cc:1467] Processing CreateTablet for tablet 379f0907bf304025b079bada3a8e9e73 (DEFAULT_TABLE table=default.Table_469367eced4344d2bbb224143a1d6e67 [id=34954b22353c44e98308a5530d6458b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.085208 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 379f0907bf304025b079bada3a8e9e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.085969 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 379f0907bf304025b079bada3a8e9e73 (DEFAULT_TABLE table=default.Table_469367eced4344d2bbb224143a1d6e67 [id=34954b22353c44e98308a5530d6458b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.086330 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 379f0907bf304025b079bada3a8e9e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.087383 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0159e9b3b87844a393d958668e318124. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.088912 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 723) took 1149 ms. Trace:
I20250115 14:52:06.089704 1751 rpcz_store.cc:276] 0115 14:52:04.939903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.939962 (+ 59us) service_pool.cc:225] Handling call
0115 14:52:05.892576 (+952614us) ts_tablet_manager.cc:1145] Deleting tablet 4abde784f9c948cb96e6cc93c30817bd
0115 14:52:05.892586 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.071367 (+178781us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.073911 (+ 2544us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.088898 (+ 14987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":952427}
I20250115 14:52:06.090795 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 724) took 1030 ms. Trace:
I20250115 14:52:06.092005 1751 rpcz_store.cc:276] 0115 14:52:05.060133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.060184 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:06.090671 (+1030487us) ts_tablet_manager.cc:1145] Deleting tablet 0a2bcfc5c5714a3eac85b6a8beafbf59
0115 14:52:06.090680 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.090778 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1028799}
I20250115 14:52:06.092075 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8af81a1841354f418cb81392cf3c252f (DEFAULT_TABLE table=default.Table_4c53ae523be9439e825de4b84817ff71 [id=66ca1d60df8040118e79a1e8ac24d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.093009 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af81a1841354f418cb81392cf3c252f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.094245 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453adba8c772446c9e24f20eed56a951. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.094341 1751 tablet_replica.cc:331] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:06.094516 1751 raft_consensus.cc:2238] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.094643 1751 raft_consensus.cc:2267] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.094085 1162 tablet_bootstrap.cc:492] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:06.095789 1162 ts_tablet_manager.cc:1397] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250115 14:52:06.097388 1751 ts_tablet_manager.cc:1905] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.102785 1162 raft_consensus.cc:357] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.103510 1162 raft_consensus.cc:383] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.103782 1162 raft_consensus.cc:738] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.104058 1168 tablet_bootstrap.cc:492] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:06.104243 1168 ts_tablet_manager.cc:1397] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.002s
I20250115 14:52:06.104306 1162 consensus_queue.cc:260] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.104790 1162 ts_tablet_manager.cc:1428] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250115 14:52:06.105044 1162 tablet_bootstrap.cc:492] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.105484 1168 raft_consensus.cc:357] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.106201 1168 raft_consensus.cc:383] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.106273 1162 tablet_bootstrap.cc:654] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.106426 1168 raft_consensus.cc:738] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
W20250115 14:52:06.106731 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4b68c168048649c5b91600eb4302b0ce consensus state has no leader
I20250115 14:52:06.106956 1168 consensus_queue.cc:260] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.107903 1168 ts_tablet_manager.cc:1428] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:52:06.108562 1168 tablet_bootstrap.cc:492] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:06.109671 1167 ts_tablet_manager.cc:1918] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.112088 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2ee5dfdef14ec092a2f9d4bfd295a6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:06.113222 1164 tablet_bootstrap.cc:492] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:06.114105 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee8b63cc21145d8993b058d8c6d23c7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.114549 776 raft_consensus.cc:1270] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250115 14:52:06.114534 984 leader_election.cc:343] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.114630 984 leader_election.cc:304] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.114863 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b85d63974ab418eb1921dd605fbf2b0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.114986 2186 raft_consensus.cc:2743] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:06.115306 984 leader_election.cc:343] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.115396 984 leader_election.cc:304] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.115703 1876 raft_consensus.cc:2743] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.116241 1751 ts_tablet_manager.cc:1918] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:06.121492 984 consensus_peers.cc:487] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83097c5bfe364418a213236435edc40c. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:06.121992 1751 log.cc:1198] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e034d7761a5247c79414b7112a1c97b2
I20250115 14:52:06.122617 1751 ts_tablet_manager.cc:1939] T e034d7761a5247c79414b7112a1c97b2 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:06.123965 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 727) took 1039 ms. Trace:
I20250115 14:52:06.114427 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2ee5dfdef14ec092a2f9d4bfd295a6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.125221 1751 rpcz_store.cc:276] 0115 14:52:05.084563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.088344 (+ 3781us) service_pool.cc:225] Handling call
0115 14:52:06.094301 (+1005957us) ts_tablet_manager.cc:1145] Deleting tablet e034d7761a5247c79414b7112a1c97b2
0115 14:52:06.094311 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.101081 (+ 6770us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.116157 (+ 15076us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.123948 (+ 7791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005770}
I20250115 14:52:06.125782 1167 log.cc:1198] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/59d8fd4b26294e8ca6745b95f2e03231
W20250115 14:52:06.113708 848 leader_election.cc:343] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.127017 2138 consensus_queue.cc:1035] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:52:06.127691 848 leader_election.cc:343] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.110146 1168 tablet_bootstrap.cc:654] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.128376 1166 ts_tablet_manager.cc:1918] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:06.128523 1166 log.cc:1198] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d5dd41d42331494e9c1b021816d0287e
I20250115 14:52:06.128918 1166 ts_tablet_manager.cc:1939] T d5dd41d42331494e9c1b021816d0287e P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:06.129474 1166 tablet_replica.cc:331] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:06.129493 1167 ts_tablet_manager.cc:1939] T 59d8fd4b26294e8ca6745b95f2e03231 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:06.129694 1166 raft_consensus.cc:2238] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.129772 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 649) took 1993 ms. Trace:
I20250115 14:52:06.129925 1166 raft_consensus.cc:2267] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.129854 1167 rpcz_store.cc:276] 0115 14:52:04.136677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.136736 (+ 59us) service_pool.cc:225] Handling call
0115 14:52:06.069100 (+1932364us) ts_tablet_manager.cc:1145] Deleting tablet 59d8fd4b26294e8ca6745b95f2e03231
0115 14:52:06.069110 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.076254 (+ 7144us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.109598 (+ 33344us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.129758 (+ 20160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3638,"tablet-delete.queue_time_us":1932213}
I20250115 14:52:06.130594 1166 ts_tablet_manager.cc:1905] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.131639 1751 tablet_replica.cc:331] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:06.132381 1751 raft_consensus.cc:2238] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.133517 1751 raft_consensus.cc:2267] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.133661 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 650) took 1984 ms. Trace:
I20250115 14:52:06.133741 1167 rpcz_store.cc:276] 0115 14:52:04.149477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.151637 (+ 2160us) service_pool.cc:225] Handling call
0115 14:52:06.133528 (+1981891us) ts_tablet_manager.cc:1145] Deleting tablet c1a77d07b1c946b18e0769a37d8a50a1
0115 14:52:06.133538 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.133643 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1981728}
I20250115 14:52:06.134522 1751 ts_tablet_manager.cc:1905] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.137482 2186 raft_consensus.cc:491] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.137599 2186 raft_consensus.cc:513] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.138190 2186 leader_election.cc:290] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.138864 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84844b5f50af408ab7dda73f7421cd9d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:06.139367 983 leader_election.cc:343] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.114492 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "104282a09c024bbaafeb5f28f91dadcb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.134088 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 655) took 1958 ms. Trace:
I20250115 14:52:06.145121 1167 rpcz_store.cc:276] 0115 14:52:04.175509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.175564 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:06.133990 (+1958426us) ts_tablet_manager.cc:1145] Deleting tablet 11fddbd9ba14443a91865387e1d1c238
0115 14:52:06.133998 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.134074 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958260}
I20250115 14:52:06.146178 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 660) took 1916 ms. Trace:
I20250115 14:52:06.146257 1167 rpcz_store.cc:276] 0115 14:52:04.229248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.229390 (+ 142us) service_pool.cc:225] Handling call
0115 14:52:06.146063 (+1916673us) ts_tablet_manager.cc:1145] Deleting tablet ed777d7496514ec5986a7e6f53d532b1
0115 14:52:06.146073 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.146160 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794082}
I20250115 14:52:06.146409 1751 ts_tablet_manager.cc:1918] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.146504 1751 log.cc:1198] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/43605c995df4400e91139199c9514bfd
I20250115 14:52:06.146785 1751 ts_tablet_manager.cc:1939] T 43605c995df4400e91139199c9514bfd P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:06.148535 1167 tablet_replica.cc:331] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:06.149139 1167 raft_consensus.cc:2238] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:06.149502 1024 tablet_service.cc:1467] Processing CreateTablet for tablet bf81e7c37c884a72bcbdb74e10ddc41b (DEFAULT_TABLE table=default.Table_ff85ca99bc044a8c8df53fe654e0cf14 [id=2c71494eee294c4e837291f8b7d7fc3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.149873 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf81e7c37c884a72bcbdb74e10ddc41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.150023 891 tablet_service.cc:1467] Processing CreateTablet for tablet ee5671e8a08d47f7bea9fc6af2f53b2d (DEFAULT_TABLE table=default.Table_96eab6cd20b842138b5acc7c8d0ac5d6 [id=816d14738e72406a88b9eed42d72be73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.150305 1167 raft_consensus.cc:2267] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.150355 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee5671e8a08d47f7bea9fc6af2f53b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.150807 875 tablet_service.cc:1467] Processing CreateTablet for tablet d7558b4d7453453f83facb0fc7e371e8 (DEFAULT_TABLE table=default.Table_ae36f7ba541a43ffad08ed6fc6b10ec0 [id=349052fac9f94319b3b24fce63bb3a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.151129 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7558b4d7453453f83facb0fc7e371e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.152522 880 tablet_service.cc:1467] Processing CreateTablet for tablet 210f161cf6b94a2abbad651c40bb8cec (DEFAULT_TABLE table=default.Table_944d3194b69f4c6a9c0899a204051442 [id=e8942d572d6f4980a22ec7ca5955801e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.152845 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 210f161cf6b94a2abbad651c40bb8cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.154251 879 tablet_service.cc:1467] Processing CreateTablet for tablet 08d9f83287c6450cb30d0d02dc1c6238 (DEFAULT_TABLE table=default.Table_28d063764bb94657a5f118c8820e8fbe [id=af5bdd7d5f044c9ba5ff703e3ee76cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.154584 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08d9f83287c6450cb30d0d02dc1c6238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.154749 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 21a3664fbeaf428b8a4f260c4b48d610 (DEFAULT_TABLE table=default.Table_28021473fc1344df97d1fdd68b2023dc [id=eedd836baa054b938d4d470bf68688c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.155043 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a3664fbeaf428b8a4f260c4b48d610. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.156332 1019 tablet_service.cc:1467] Processing CreateTablet for tablet ee5671e8a08d47f7bea9fc6af2f53b2d (DEFAULT_TABLE table=default.Table_96eab6cd20b842138b5acc7c8d0ac5d6 [id=816d14738e72406a88b9eed42d72be73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.156628 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee5671e8a08d47f7bea9fc6af2f53b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.150820 1167 ts_tablet_manager.cc:1905] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.160526 1011 tablet_service.cc:1467] Processing CreateTablet for tablet c3cb017511424c1f94fdfc02ad2b57b1 (DEFAULT_TABLE table=default.Table_a69fe39cbd8f4a628213badbaafff14a [id=c250cbb13ef148b29f2bcf7ba8b8125b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.160831 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3cb017511424c1f94fdfc02ad2b57b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.163727 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 210f161cf6b94a2abbad651c40bb8cec (DEFAULT_TABLE table=default.Table_944d3194b69f4c6a9c0899a204051442 [id=e8942d572d6f4980a22ec7ca5955801e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.163916 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d7558b4d7453453f83facb0fc7e371e8 (DEFAULT_TABLE table=default.Table_ae36f7ba541a43ffad08ed6fc6b10ec0 [id=349052fac9f94319b3b24fce63bb3a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.164153 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 210f161cf6b94a2abbad651c40bb8cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.164399 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7558b4d7453453f83facb0fc7e371e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.164678 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 08d9f83287c6450cb30d0d02dc1c6238 (DEFAULT_TABLE table=default.Table_28d063764bb94657a5f118c8820e8fbe [id=af5bdd7d5f044c9ba5ff703e3ee76cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.165019 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08d9f83287c6450cb30d0d02dc1c6238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.167982 882 tablet_service.cc:1467] Processing CreateTablet for tablet c3cb017511424c1f94fdfc02ad2b57b1 (DEFAULT_TABLE table=default.Table_a69fe39cbd8f4a628213badbaafff14a [id=c250cbb13ef148b29f2bcf7ba8b8125b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.168987 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3cb017511424c1f94fdfc02ad2b57b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.170351 879 tablet_service.cc:1467] Processing CreateTablet for tablet 2a005ad11e7941fab96b5565d55cfaf7 (DEFAULT_TABLE table=default.Table_28ceb002e729415a92c24a593b1ba77a [id=97ab47b613954298b39ace13c5a2b076]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.170747 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a005ad11e7941fab96b5565d55cfaf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.173539 2138 raft_consensus.cc:491] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.173664 2138 raft_consensus.cc:513] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.173895 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 2a005ad11e7941fab96b5565d55cfaf7 (DEFAULT_TABLE table=default.Table_28ceb002e729415a92c24a593b1ba77a [id=97ab47b613954298b39ace13c5a2b076]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.174175 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a005ad11e7941fab96b5565d55cfaf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.174281 2138 leader_election.cc:290] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:06.176728 847 leader_election.cc:343] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: fc3d2ca7cd7c48a79ed1396fbbdc6866
I20250115 14:52:06.148435 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84844b5f50af408ab7dda73f7421cd9d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:06.178738 984 leader_election.cc:343] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.178849 984 leader_election.cc:304] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.179052 2186 raft_consensus.cc:2743] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.183043 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d195be2d543f4a8693045a024912c43a (table default.Table_bba588d115d34226951843abdb9b6ebe [id=e99d33d2241343e2818d4217babf9a49]) successfully deleted
I20250115 14:52:06.188120 893 tablet_service.cc:1467] Processing CreateTablet for tablet bf81e7c37c884a72bcbdb74e10ddc41b (DEFAULT_TABLE table=default.Table_ff85ca99bc044a8c8df53fe654e0cf14 [id=2c71494eee294c4e837291f8b7d7fc3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.183763 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1cc96e57499c4c8ba91f9724a57dadbd" } new_table_name: "default.Table_b3e4daa02d5e4d8ca7f0710e6c5f8260" modify_external_catalogs: true
I20250115 14:52:06.184008 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3e03f479683449e197a5150099b49504"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:06.184176 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_572841278ecd4e1d9c55e42ea357b3c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.189958 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e03f479683449e197a5150099b49504 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.184549 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b02e318b61ed4745bf6bf686cd720090"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.190531 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b02e318b61ed4745bf6bf686cd720090 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:06.189958 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_572841278ecd4e1d9c55e42ea357b3c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.184895 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f047206807ce42eb89594d08a8209682"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.191805 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f047206807ce42eb89594d08a8209682 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.184561 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7c155cee1f6e49a99d8ab38956a737e2" } modify_external_catalogs: true
I20250115 14:52:06.192412 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7c155cee1f6e49a99d8ab38956a737e2" } modify_external_catalogs: true
W20250115 14:52:06.195523 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e180e37a1d5d46d78380323662b80f85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e180e37a1d5d46d78380323662b80f85 already in progress: creating tablet
I20250115 14:52:06.200933 1751 tablet_replica.cc:331] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:06.202565 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 59d8fd4b26294e8ca6745b95f2e03231 (table default.Table_98cb517939064897baf89d74e6e0355e [id=757cda13648a4e3d9b02d70555966106]) successfully deleted
I20250115 14:52:06.204708 2079 tablet.cc:1721] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:06.206312 886 tablet_service.cc:1467] Processing CreateTablet for tablet 21a3664fbeaf428b8a4f260c4b48d610 (DEFAULT_TABLE table=default.Table_28021473fc1344df97d1fdd68b2023dc [id=eedd836baa054b938d4d470bf68688c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.212075 1162 tablet_bootstrap.cc:492] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:06.212206 1162 ts_tablet_manager.cc:1397] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.009s
I20250115 14:52:06.212870 1162 raft_consensus.cc:357] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.213080 1162 raft_consensus.cc:383] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.213132 1162 raft_consensus.cc:738] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.213308 1162 consensus_queue.cc:260] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.214021 1162 ts_tablet_manager.cc:1428] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:06.214293 1162 tablet_bootstrap.cc:492] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.215619 1162 tablet_bootstrap.cc:654] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.216470 1751 raft_consensus.cc:2238] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.216630 1751 raft_consensus.cc:2267] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.211328 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]) successfully deleted
I20250115 14:52:06.217113 1751 ts_tablet_manager.cc:1905] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:06.217615 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c1a77d07b1c946b18e0769a37d8a50a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1a77d07b1c946b18e0769a37d8a50a1 already in progress: creating tablet
I20250115 14:52:06.220557 2149 tablet.cc:1721] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250115 14:52:06.221899 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99449d50c2dd46159a69dc0af1de7a87 consensus state has no leader
I20250115 14:52:06.225049 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:06.225493 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4ee8b63cc21145d8993b058d8c6d23c7 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]). Expected version 1 got 0
I20250115 14:52:06.226344 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a3664fbeaf428b8a4f260c4b48d610. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.226579 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet abccc613e1754839a4c7a8f5c448a852
I20250115 14:52:06.229465 447 catalog_manager.cc:6596] ReplaceTablet: tablet abccc613e1754839a4c7a8f5c448a852 deleted and replaced by tablet 11aeb6c73b4d42058736ffce551e311c
I20250115 14:52:06.231884 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f8719599d09f457681ab1c5bbf617baa" } modify_external_catalogs: true
I20250115 14:52:06.232174 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f8719599d09f457681ab1c5bbf617baa" } modify_external_catalogs: true
I20250115 14:52:06.235298 1751 ts_tablet_manager.cc:1918] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:06.241144 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 11fddbd9ba14443a91865387e1d1c238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11fddbd9ba14443a91865387e1d1c238 already in progress: creating tablet
I20250115 14:52:06.242918 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf81e7c37c884a72bcbdb74e10ddc41b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:06.243721 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ed777d7496514ec5986a7e6f53d532b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed777d7496514ec5986a7e6f53d532b1 already in progress: creating tablet
I20250115 14:52:06.244709 1166 ts_tablet_manager.cc:1918] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.252559 1751 log.cc:1198] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/009ddfe9dfe14ba2b067f907a9d13538
I20250115 14:52:06.253044 1751 ts_tablet_manager.cc:1939] T 009ddfe9dfe14ba2b067f907a9d13538 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:06.260710 1751 tablet_replica.cc:331] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:06.261008 1751 raft_consensus.cc:2238] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.261245 1751 raft_consensus.cc:2267] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.261895 1751 ts_tablet_manager.cc:1905] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.267354 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 4abde784f9c948cb96e6cc93c30817bd (table default.Table_10deaf92a667422fa029a731f755dfef [id=e657e05ca35045f1a196dd012ee59483]) successfully deleted
I20250115 14:52:06.281018 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b1ed6d26296a4f9799945452887c3c1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.286010 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1ed6d26296a4f9799945452887c3c1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:06.281287 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92aeca346a7746468dbbff053d933472 consensus state has no leader
I20250115 14:52:06.288512 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_050541babd7b40668d93e38f18bef277" } new_table_name: "default.Table_f459a15fa0234ba9b5722b7bbff3b6cb" modify_external_catalogs: true
I20250115 14:52:06.281764 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d8d3e6a881ce475a8f1319238ab8c610" } modify_external_catalogs: true
I20250115 14:52:06.290007 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d8d3e6a881ce475a8f1319238ab8c610" } modify_external_catalogs: true
I20250115 14:52:06.282223 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2179cfd883084fe7864e4cf7bee57581"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.290899 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2179cfd883084fe7864e4cf7bee57581 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.282624 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a75e911886994dffbae81cf1e8e94849"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:06.283151 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c14836b97c3247b39ff9072c12e642e0
I20250115 14:52:06.292663 1751 ts_tablet_manager.cc:1918] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:06.292860 1751 log.cc:1198] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/dff155a5c2b0479a90e74993de6f402d
I20250115 14:52:06.293103 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e37f4580bc3c4d5cacb53a430d3d5605"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:06.293248 1751 ts_tablet_manager.cc:1939] T dff155a5c2b0479a90e74993de6f402d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:06.293536 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e37f4580bc3c4d5cacb53a430d3d5605 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.294106 2186 raft_consensus.cc:491] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.294224 2186 raft_consensus.cc:513] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.283566 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_28ceb002e729415a92c24a593b1ba77a" } new_table_name: "default.Table_c71aa4f14fe849e9be7101f2660dcfee" modify_external_catalogs: true
I20250115 14:52:06.294847 2186 leader_election.cc:290] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.296599 1751 tablet_replica.cc:331] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:06.296831 1751 raft_consensus.cc:2238] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.297019 1751 raft_consensus.cc:2267] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.297773 1751 ts_tablet_manager.cc:1905] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:06.299728 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a75e911886994dffbae81cf1e8e94849 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.283908 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f60e68b411364ada9b89879012d862ea
I20250115 14:52:06.303479 1751 ts_tablet_manager.cc:1918] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.303561 1751 log.cc:1198] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/5b85d63974ab418eb1921dd605fbf2b0
I20250115 14:52:06.303805 1751 ts_tablet_manager.cc:1939] T 5b85d63974ab418eb1921dd605fbf2b0 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:06.284067 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91] alter complete (version 3)
I20250115 14:52:06.305902 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f0d5f1bc94b941bf9f5d55369bdfe2ef" } new_table_name: "default.Table_58c17e0ad22e41738519c725c25dcdfc" modify_external_catalogs: true
W20250115 14:52:06.285010 448 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 4ee8b63cc21145d8993b058d8c6d23c7 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 4ee8b63cc21145d8993b058d8c6d23c7 consensus state has no leader
I20250115 14:52:06.309583 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0b05d8cc5e384fccb6f6949ae2c33c98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:06.309430 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ff33645127a443efb2b776d808942dba" } modify_external_catalogs: true
I20250115 14:52:06.310575 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ff33645127a443efb2b776d808942dba" } modify_external_catalogs: true
W20250115 14:52:06.311466 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b05d8cc5e384fccb6f6949ae2c33c98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:06.275686 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 already in progress: creating tablet
I20250115 14:52:06.313477 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e034d7761a5247c79414b7112a1c97b2 (table default.Table_4deac3b7f7124b6aa49514a17811b1c3 [id=efb16fff4d6c4f44af7dfaaa00d4f444]) successfully deleted
I20250115 14:52:06.251255 1167 ts_tablet_manager.cc:1918] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:06.315347 1167 log.cc:1198] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/a915e0c0a30940ed8a6233f117e2a560
I20250115 14:52:06.315737 1167 ts_tablet_manager.cc:1939] T a915e0c0a30940ed8a6233f117e2a560 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:06.316064 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 675) took 1847 ms. Trace:
I20250115 14:52:06.316138 1167 rpcz_store.cc:276] 0115 14:52:04.468223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.468823 (+ 600us) service_pool.cc:225] Handling call
0115 14:52:06.148486 (+1679663us) ts_tablet_manager.cc:1145] Deleting tablet a915e0c0a30940ed8a6233f117e2a560
0115 14:52:06.148497 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.226884 (+ 78387us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.251174 (+ 24290us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.316047 (+ 64873us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1679508}
I20250115 14:52:06.316531 1167 tablet_replica.cc:331] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:06.316730 1167 raft_consensus.cc:2238] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.316885 1167 raft_consensus.cc:2267] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.317435 1167 ts_tablet_manager.cc:1905] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:06.315044 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 57aecd22f97c41618f90d14c2f53a7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57aecd22f97c41618f90d14c2f53a7c9 already in progress: creating tablet
W20250115 14:52:06.319343 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 65a6f83730f74d13823bd15d93538df7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65a6f83730f74d13823bd15d93538df7 already in progress: creating tablet
W20250115 14:52:06.320115 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d1259256b7744e739581125611abbe67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1259256b7744e739581125611abbe67 already in progress: creating tablet
I20250115 14:52:06.264925 2202 raft_consensus.cc:491] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.320403 2202 raft_consensus.cc:513] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.321095 2202 leader_election.cc:290] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.265539 1168 tablet_bootstrap.cc:492] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:06.321291 1168 ts_tablet_manager.cc:1397] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.213s user 0.005s sys 0.007s
I20250115 14:52:06.321985 1168 raft_consensus.cc:357] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.322150 1168 raft_consensus.cc:383] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.322202 1168 raft_consensus.cc:738] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.322379 1168 consensus_queue.cc:260] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.322902 1168 ts_tablet_manager.cc:1428] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250115 14:52:06.295491 848 leader_election.cc:343] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.295517 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8442dc919d5f4d63b9b2ce7276cc0bc5" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:06.323015 848 leader_election.cc:304] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.323557 2138 raft_consensus.cc:2743] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.327301 1168 tablet_bootstrap.cc:492] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:06.327339 983 leader_election.cc:343] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.327621 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f9262e808b4589a21257ed3fb0363b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:06.327823 1049 raft_consensus.cc:2463] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:06.329046 1168 tablet_bootstrap.cc:654] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.311687 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ca5ad5415b9144e583b4c771d24d4431" } modify_external_catalogs: true
I20250115 14:52:06.330271 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ca5ad5415b9144e583b4c771d24d4431" } modify_external_catalogs: true
I20250115 14:52:06.332204 1166 log.cc:1198] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/5bc975b044dd439dbd1f8f9a82cce751
I20250115 14:52:06.329982 847 leader_election.cc:304] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:06.332587 2202 raft_consensus.cc:2798] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:06.332635 1166 ts_tablet_manager.cc:1939] T 5bc975b044dd439dbd1f8f9a82cce751 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:06.332653 2202 raft_consensus.cc:491] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:06.332705 2202 raft_consensus.cc:3054] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.332871 2200 raft_consensus.cc:491] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.332950 2200 raft_consensus.cc:513] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.248855 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 010e6535b6bc44f391e6f2b6f1686532 with delete_type TABLET_DATA_DELETED (replaced by tablet 95d891d568574c51b0d062ca9ac77f45) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.289614 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:06.289624 2131 tablet.cc:1721] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:06.289975 2225 raft_consensus.cc:491] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.334848 2225 raft_consensus.cc:513] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:06.335014 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3df4c6d310d7465ca769a5b3a13b8f59 consensus state has no leader
I20250115 14:52:06.335747 2225 leader_election.cc:290] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.298682 752 tablet_service.cc:1467] Processing CreateTablet for tablet 379f0907bf304025b079bada3a8e9e73 (DEFAULT_TABLE table=default.Table_469367eced4344d2bbb224143a1d6e67 [id=34954b22353c44e98308a5530d6458b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.336071 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 379f0907bf304025b079bada3a8e9e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.337221 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1]). Expected version 3 got 0
I20250115 14:52:06.337567 454 catalog_manager.cc:5526] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:06.337957 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_4b79de2dd7fe493f84a77ae3c477e2c8 [id=471b029f1dba4dd087deb339bba02013]). Expected version 1 got 0
I20250115 14:52:06.299363 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3d2ca7cd7c48a79ed1396fbbdc6866" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.295867 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8442dc919d5f4d63b9b2ce7276cc0bc5" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:06.340770 984 leader_election.cc:343] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.340879 984 leader_election.cc:304] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.341280 2186 raft_consensus.cc:2743] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:06.342470 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a005ad11e7941fab96b5565d55cfaf7 consensus state has no leader
W20250115 14:52:06.343115 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet fb81e8fcc38a41e69cfb0e34dbaaac84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb81e8fcc38a41e69cfb0e34dbaaac84 already in progress: creating tablet
I20250115 14:52:06.344498 744 tablet_service.cc:1467] Processing CreateTablet for tablet 21a3664fbeaf428b8a4f260c4b48d610 (DEFAULT_TABLE table=default.Table_28021473fc1344df97d1fdd68b2023dc [id=eedd836baa054b938d4d470bf68688c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.344550 751 tablet_service.cc:1467] Processing CreateTablet for tablet ee5671e8a08d47f7bea9fc6af2f53b2d (DEFAULT_TABLE table=default.Table_96eab6cd20b842138b5acc7c8d0ac5d6 [id=816d14738e72406a88b9eed42d72be73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.344888 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a3664fbeaf428b8a4f260c4b48d610. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.344928 748 tablet_service.cc:1467] Processing CreateTablet for tablet c3cb017511424c1f94fdfc02ad2b57b1 (DEFAULT_TABLE table=default.Table_a69fe39cbd8f4a628213badbaafff14a [id=c250cbb13ef148b29f2bcf7ba8b8125b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.345484 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3cb017511424c1f94fdfc02ad2b57b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.348380 2138 raft_consensus.cc:491] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.348508 2138 raft_consensus.cc:513] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.348860 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee5671e8a08d47f7bea9fc6af2f53b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.349035 2200 leader_election.cc:290] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.352000 2138 leader_election.cc:290] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:06.352973 847 leader_election.cc:343] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d71a1ecd900942e5ae788c9a28c70a7a
I20250115 14:52:06.354993 745 tablet_service.cc:1467] Processing CreateTablet for tablet bf81e7c37c884a72bcbdb74e10ddc41b (DEFAULT_TABLE table=default.Table_ff85ca99bc044a8c8df53fe654e0cf14 [id=2c71494eee294c4e837291f8b7d7fc3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.355294 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf81e7c37c884a72bcbdb74e10ddc41b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.356509 752 tablet_service.cc:1467] Processing CreateTablet for tablet 8af81a1841354f418cb81392cf3c252f (DEFAULT_TABLE table=default.Table_4c53ae523be9439e825de4b84817ff71 [id=66ca1d60df8040118e79a1e8ac24d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.356814 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af81a1841354f418cb81392cf3c252f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.360201 1167 ts_tablet_manager.cc:1918] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.360384 1167 log.cc:1198] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/37ad2b7350294375bd30fb4b46906491
I20250115 14:52:06.360723 1167 ts_tablet_manager.cc:1939] T 37ad2b7350294375bd30fb4b46906491 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:06.361182 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 676) took 1887 ms. Trace:
I20250115 14:52:06.361470 1167 rpcz_store.cc:276] 0115 14:52:04.473603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.474113 (+ 510us) service_pool.cc:225] Handling call
0115 14:52:06.316485 (+1842372us) ts_tablet_manager.cc:1145] Deleting tablet 37ad2b7350294375bd30fb4b46906491
0115 14:52:06.316496 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.329540 (+ 13044us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.360120 (+ 30580us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.361168 (+ 1048us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1842215}
I20250115 14:52:06.362126 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 680) took 1778 ms. Trace:
I20250115 14:52:06.362841 1167 rpcz_store.cc:276] 0115 14:52:04.583314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.583345 (+ 31us) service_pool.cc:225] Handling call
0115 14:52:06.362008 (+1778663us) ts_tablet_manager.cc:1145] Deleting tablet 282d988bc41443c381bb0ba69b618f5d
0115 14:52:06.362018 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.362112 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1778502}
W20250115 14:52:06.365557 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet be8e12ccb8d14c1c9258dc3f81d1527d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be8e12ccb8d14c1c9258dc3f81d1527d already in progress: creating tablet
I20250115 14:52:06.370704 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 682) took 1787 ms. Trace:
I20250115 14:52:06.370788 1167 rpcz_store.cc:276] 0115 14:52:04.583507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.583573 (+ 66us) service_pool.cc:225] Handling call
0115 14:52:06.370596 (+1787023us) ts_tablet_manager.cc:1145] Deleting tablet 74d00bcee22d40769102eacaf728480d
0115 14:52:06.370605 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.370687 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1786890}
W20250115 14:52:06.372468 847 leader_election.cc:343] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 7d07686c40324233a49e4c49aeed73f6
I20250115 14:52:06.374361 1162 tablet_bootstrap.cc:492] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:06.374577 1162 ts_tablet_manager.cc:1397] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.006s
I20250115 14:52:06.375361 1162 raft_consensus.cc:357] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.378422 2202 raft_consensus.cc:513] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.378988 2202 leader_election.cc:290] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.379271 1164 ts_tablet_manager.cc:1397] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.464s user 0.016s sys 0.025s
I20250115 14:52:06.379778 1162 raft_consensus.cc:383] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.380028 1162 raft_consensus.cc:738] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.380120 744 tablet_service.cc:1467] Processing CreateTablet for tablet 08d9f83287c6450cb30d0d02dc1c6238 (DEFAULT_TABLE table=default.Table_28d063764bb94657a5f118c8820e8fbe [id=af5bdd7d5f044c9ba5ff703e3ee76cf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.380509 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08d9f83287c6450cb30d0d02dc1c6238. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.381024 1162 consensus_queue.cc:260] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.384738 748 tablet_service.cc:1467] Processing CreateTablet for tablet d7558b4d7453453f83facb0fc7e371e8 (DEFAULT_TABLE table=default.Table_ae36f7ba541a43ffad08ed6fc6b10ec0 [id=349052fac9f94319b3b24fce63bb3a70]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.385125 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7558b4d7453453f83facb0fc7e371e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.386137 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:06.388686 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2a005ad11e7941fab96b5565d55cfaf7 (DEFAULT_TABLE table=default.Table_28ceb002e729415a92c24a593b1ba77a [id=97ab47b613954298b39ace13c5a2b076]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.389048 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a005ad11e7941fab96b5565d55cfaf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.392135 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 683) took 1808 ms. Trace:
I20250115 14:52:06.392225 1167 rpcz_store.cc:276] 0115 14:52:04.583596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.583650 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:06.391973 (+1808323us) ts_tablet_manager.cc:1145] Deleting tablet 9494c0c57be8499a883888d067c7db77
0115 14:52:06.391982 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.392116 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808157}
I20250115 14:52:06.392693 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 679) took 1809 ms. Trace:
I20250115 14:52:06.392769 1167 rpcz_store.cc:276] 0115 14:52:04.583211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.583766 (+ 555us) service_pool.cc:225] Handling call
0115 14:52:06.392602 (+1808836us) ts_tablet_manager.cc:1145] Deleting tablet 398ac8af619c427287b3964443173864
0115 14:52:06.392610 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.392679 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808655}
I20250115 14:52:06.393162 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 681) took 1809 ms. Trace:
I20250115 14:52:06.394385 746 tablet_service.cc:1467] Processing CreateTablet for tablet d849c4e268a54191ae2632884f61edc0 (DEFAULT_TABLE table=default.Table_c4f5e92e6f994cb0ad035e3cef60b3ce [id=9ed3557619724e11b7875953397cee2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.394685 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d849c4e268a54191ae2632884f61edc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:06.397074 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 567365040b6d4d6eba04112c8d4c3dfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567365040b6d4d6eba04112c8d4c3dfb already in progress: creating tablet
I20250115 14:52:06.400461 751 tablet_service.cc:1467] Processing CreateTablet for tablet 210f161cf6b94a2abbad651c40bb8cec (DEFAULT_TABLE table=default.Table_944d3194b69f4c6a9c0899a204051442 [id=e8942d572d6f4980a22ec7ca5955801e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.400767 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 210f161cf6b94a2abbad651c40bb8cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.400187 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 43605c995df4400e91139199c9514bfd (table default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]) successfully deleted
I20250115 14:52:06.400732 1162 ts_tablet_manager.cc:1428] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20250115 14:52:06.403412 1166 tablet_replica.cc:331] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:06.403615 1166 raft_consensus.cc:2238] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.403793 1166 raft_consensus.cc:2267] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.401923 1751 tablet_replica.cc:331] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:06.409255 2149 tablet.cc:1721] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:06.409574 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 5bc975b044dd439dbd1f8f9a82cce751 (table default.Table_124bdb85e4194ff5a214a6b79870a1cf [id=f8ea853486b54a39843ef2b5628fbe37]) successfully deleted
I20250115 14:52:06.409848 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010e6535b6bc44f391e6f2b6f1686532" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:06.410044 908 raft_consensus.cc:2463] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:06.410179 2079 tablet.cc:1721] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250115 14:52:06.410382 848 leader_election.cc:343] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.410480 848 leader_election.cc:304] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.406054 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad3cda1056349a789a1751c5d386d0c (DEFAULT_TABLE table=default.Table_b28114e5556947c4aaba883427f90c2a [id=2d52564d7100489c803c031d747f6f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.412386 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad3cda1056349a789a1751c5d386d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.413549 2200 raft_consensus.cc:2743] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.406199 756 tablet_service.cc:1467] Processing CreateTablet for tablet 79736602ea2646b39708ca9c6834e847 (DEFAULT_TABLE table=default.Table_1e83e61b373d4455b007c7214a0dba53 [id=92d264b71c3a4716b8d930d7b8677f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.416651 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79736602ea2646b39708ca9c6834e847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.417899 755 tablet_service.cc:1467] Processing CreateTablet for tablet 75ccc1f31c6a4d2cba65324b7a71c2ce (DEFAULT_TABLE table=default.Table_55bce736d0234efa9eeee2a1745b955a [id=67af6a5e607747ac8e6831dc2051b1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.418164 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ccc1f31c6a4d2cba65324b7a71c2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.419530 1096 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:06.406360 749 tablet_service.cc:1467] Processing CreateTablet for tablet 9c35882d2e5c464c9cfad496ba9d6af9 (DEFAULT_TABLE table=default.Table_958850c3f0a54c49a8dd42b180068897 [id=8242d7a08fcd4d4e836f6b70b1bacdc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.420650 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c35882d2e5c464c9cfad496ba9d6af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.423192 755 tablet_service.cc:1467] Processing CreateTablet for tablet 93e377a771a0427db8fe8135cdaa1ff3 (DEFAULT_TABLE table=default.Table_679ebe5f3b29426082b31ce8a77da689 [id=da2f0bc60826498fabaf63c5ece311ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.423538 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e377a771a0427db8fe8135cdaa1ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.407016 741 tablet_service.cc:1467] Processing CreateTablet for tablet 546fe492182f47b7b008f4719f97e877 (DEFAULT_TABLE table=default.Table_ba297fd1bf8c47dabb7b1dab458cf8df [id=d0926ed2b8324e67bbcf6ae4b4a2154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.424621 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 546fe492182f47b7b008f4719f97e877. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.404270 1166 ts_tablet_manager.cc:1905] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.407140 750 tablet_service.cc:1514] Processing DeleteTablet for tablet c14836b97c3247b39ff9072c12e642e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.428076 1162 tablet_bootstrap.cc:492] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.428505 744 tablet_service.cc:1467] Processing CreateTablet for tablet 99449d50c2dd46159a69dc0af1de7a87 (DEFAULT_TABLE table=default.Table_201c8e612d7d4a85a6d4d4b0a5095254 [id=b74194f7d2f547cdad4e1e59ddec89d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.428591 750 tablet_service.cc:1467] Processing CreateTablet for tablet f1e3879802464acaa279dfe4fbb634bb (DEFAULT_TABLE table=default.Table_b84df010f86642d4b3585c0e774575bf [id=1d71f4010b0f4682ab1a107c0806129b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.406677 740 tablet_service.cc:1467] Processing CreateTablet for tablet bc1440e3cd444cf8b3365a0af8de8958 (DEFAULT_TABLE table=default.Table_6f0b6c724c1c43e7adf538ad7369b7a1 [id=9e8ce51e784e4bdbb3ac78cba10bfedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.428166 1751 raft_consensus.cc:2238] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.407840 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9449e1d09c3148ee86074a57c8e22240 (DEFAULT_TABLE table=default.Table_78ff869a9d0c483a87b4d43f8bcac7c9 [id=629bac9d176a4083ab706748a9a7252e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.407649 747 tablet_service.cc:1467] Processing CreateTablet for tablet 792c68ba824640c1b653ad0bfbe2de02 (DEFAULT_TABLE table=default.Table_21e00049c7364adb9a87e33cd7a04957 [id=94f032d2cb6c42f9aea0538a13bed768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.408064 737 tablet_service.cc:1467] Processing CreateTablet for tablet 3df4c6d310d7465ca769a5b3a13b8f59 (DEFAULT_TABLE table=default.Table_f459a15fa0234ba9b5722b7bbff3b6cb [id=a5a9ecce138e4b7aa6a792e00d79d997]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.430521 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7b6ea13ef1884d66bd1e7655df3e925d (DEFAULT_TABLE table=default.Table_380c0dd6df914022a741ea83ed384911 [id=190589a22d424365a7f858a214fd0f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.431494 745 tablet_service.cc:1467] Processing CreateTablet for tablet 4e39e13fb8c84ae18bac5e45d576a0ab (DEFAULT_TABLE table=default.Table_81803594c5c741619ae708256092f827 [id=caa6adb7563c41088e5120993c46c54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.432629 1751 raft_consensus.cc:2267] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.434016 741 tablet_service.cc:1467] Processing CreateTablet for tablet 1df001f0c00149609e20cf9dd049e8b5 (DEFAULT_TABLE table=default.Table_14622d246cb34c0aa0c8527eebb4319b [id=054f4d9637c94374b720f86b1c96d509]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.434808 746 tablet_service.cc:1467] Processing CreateTablet for tablet 026d05f8fb584da39000d83546b09e63 (DEFAULT_TABLE table=default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.438539 749 tablet_service.cc:1467] Processing CreateTablet for tablet 064f890830304758a6840a1502e59080 (DEFAULT_TABLE table=default.Table_adcff7acc18b46e79d647e28af423fbd [id=d6f2abb6e8894bf58f3aa4876cc38ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.406981 739 tablet_service.cc:1514] Processing DeleteTablet for tablet abccc613e1754839a4c7a8f5c448a852 with delete_type TABLET_DATA_DELETED (replaced by tablet 11aeb6c73b4d42058736ffce551e311c) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.405608 743 tablet_service.cc:1467] Processing CreateTablet for tablet eba5819df2944a84a636c6934b18510d (DEFAULT_TABLE table=default.Table_352831c0b6b04ca6825725b7825ecb12 [id=39c963a5bc8f4a3b9ff5780bb75e33e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.439433 739 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa67705596742c6b942f49bca0aa817 (DEFAULT_TABLE table=default.Table_3b9fd4f4e09a40e99f015f099eef8306 [id=254935fbfd31462c95a7762ab90fff86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.407428 742 tablet_service.cc:1467] Processing CreateTablet for tablet f3be5fb895424a3498c22d0108034ad1 (DEFAULT_TABLE table=default.Table_38055c6c2fc748eb9fcefff19f2e92e9 [id=63588086fdf344819a066c04501cab6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.407683 738 tablet_service.cc:1467] Processing CreateTablet for tablet b126874bae6e4c84bd97f5ff65e0882b (DEFAULT_TABLE table=default.Table_340265c995a649808c582e4c4e6d0b22 [id=bf2a1ee254eb411dbbaccc0a78dfbad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.407217 754 tablet_service.cc:1514] Processing DeleteTablet for tablet f60e68b411364ada9b89879012d862ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.428807 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99449d50c2dd46159a69dc0af1de7a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.441111 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e3879802464acaa279dfe4fbb634bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.441254 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b126874bae6e4c84bd97f5ff65e0882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.443588 1751 ts_tablet_manager.cc:1905] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.444379 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3be5fb895424a3498c22d0108034ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.448671 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa67705596742c6b942f49bca0aa817. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.449167 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 792c68ba824640c1b653ad0bfbe2de02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.452389 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df4c6d310d7465ca769a5b3a13b8f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.452864 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ae36f7ba541a43ffad08ed6fc6b10ec0" } new_table_name: "default.Table_4e2c0c1386cf461b89310fa41aa7a4d4" modify_external_catalogs: true
I20250115 14:52:06.456398 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc1440e3cd444cf8b3365a0af8de8958. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:06.456833 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet db5518ac647c472bb47ed741022c820a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db5518ac647c472bb47ed741022c820a already in progress: creating tablet
I20250115 14:52:06.459689 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eba5819df2944a84a636c6934b18510d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.460378 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9449e1d09c3148ee86074a57c8e22240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.461019 1162 tablet_bootstrap.cc:654] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.461089 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 009ddfe9dfe14ba2b067f907a9d13538 (table default.Table_96eab6cd20b842138b5acc7c8d0ac5d6 [id=816d14738e72406a88b9eed42d72be73]) successfully deleted
W20250115 14:52:06.465171 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a600fa0f0ec049dca2803bf7d2c74a95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a600fa0f0ec049dca2803bf7d2c74a95 already in progress: creating tablet
I20250115 14:52:06.467693 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026d05f8fb584da39000d83546b09e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.470357 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e39e13fb8c84ae18bac5e45d576a0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.472424 2200 consensus_queue.cc:1035] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:06.475457 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b6ea13ef1884d66bd1e7655df3e925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.481293 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df001f0c00149609e20cf9dd049e8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.484439 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 064f890830304758a6840a1502e59080. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.488590 2202 consensus_queue.cc:1035] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:06.492977 2186 raft_consensus.cc:491] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.493110 2186 raft_consensus.cc:513] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.493747 2186 leader_election.cc:290] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.494422 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a059e2f0654a4aafe9e8082b8fde7a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:06.494983 1166 ts_tablet_manager.cc:1918] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:06.495039 983 leader_election.cc:343] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.495270 1166 log.cc:1198] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/009ddfe9dfe14ba2b067f907a9d13538
I20250115 14:52:06.495757 1166 ts_tablet_manager.cc:1939] T 009ddfe9dfe14ba2b067f907a9d13538 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:06.495910 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a059e2f0654a4aafe9e8082b8fde7a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.496140 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f9262e808b4589a21257ed3fb0363b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:06.496374 1049 raft_consensus.cc:3054] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:06.497082 984 leader_election.cc:343] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.497180 984 leader_election.cc:304] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.497601 2186 raft_consensus.cc:2743] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.498842 1049 raft_consensus.cc:2463] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:06.499257 847 leader_election.cc:304] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:06.499583 2202 raft_consensus.cc:2798] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:06.499832 2202 raft_consensus.cc:695] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:06.500134 2202 consensus_queue.cc:237] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.410991 748 tablet_service.cc:1467] Processing CreateTablet for tablet 36749313f40648ac8cec4ac7f3d97cf8 (DEFAULT_TABLE table=default.Table_7f8c179068074379b0e5baec32f2699b [id=b1b61ad1a148420585b4d68b675f2055]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:06.512483 710 leader_election.cc:343] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 010e6535b6bc44f391e6f2b6f1686532
I20250115 14:52:06.513589 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d07686c40324233a49e4c49aeed73f6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.514076 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71a1ecd900942e5ae788c9a28c70a7a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.514501 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f9262e808b4589a21257ed3fb0363b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:06.515710 2131 tablet.cc:1721] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:06.526157 2059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=484 overwritten=0 applied=484 ignored=0} inserts{seen=446 ignored=0} mutations{seen=559 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250115 14:52:06.526836 2059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: Bootstrap complete.
I20250115 14:52:06.530442 1843 raft_consensus.cc:491] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.530570 1843 raft_consensus.cc:513] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.531234 1843 leader_election.cc:290] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.531919 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68c168048649c5b91600eb4302b0ce" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:06.532119 1048 raft_consensus.cc:2463] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:06.532433 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f9262e808b4589a21257ed3fb0363b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.532896 847 leader_election.cc:304] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:06.533329 1843 raft_consensus.cc:2798] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:06.533406 1843 raft_consensus.cc:491] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:06.533468 1843 raft_consensus.cc:3054] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.535873 1843 raft_consensus.cc:513] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.540908 1843 leader_election.cc:290] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.541307 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68c168048649c5b91600eb4302b0ce" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:06.541478 1048 raft_consensus.cc:3054] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.543954 1048 raft_consensus.cc:2463] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:06.544507 847 leader_election.cc:304] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:06.545555 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36749313f40648ac8cec4ac7f3d97cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.545691 1164 raft_consensus.cc:357] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.545869 1164 raft_consensus.cc:383] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.545946 1164 raft_consensus.cc:738] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.546132 1164 consensus_queue.cc:260] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.546674 1164 ts_tablet_manager.cc:1428] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.167s user 0.002s sys 0.002s
I20250115 14:52:06.547250 2200 raft_consensus.cc:2798] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:06.547362 2200 raft_consensus.cc:695] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:06.547861 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68c168048649c5b91600eb4302b0ce" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.548022 772 raft_consensus.cc:2463] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:06.548413 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b68c168048649c5b91600eb4302b0ce" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:06.547590 2059 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:52:06.548538 772 raft_consensus.cc:3054] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.548686 2059 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d97cf5f1fbb4bc6883711020e2fa3f3, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.549434 2059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 484, Last appended: 1.484, Last appended by leader: 484, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
W20250115 14:52:06.549670 847 consensus_peers.cc:487] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dff155a5c2b0479a90e74993de6f402d. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:06.550805 772 raft_consensus.cc:2463] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
W20250115 14:52:06.551935 848 leader_election.cc:343] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.552043 848 leader_election.cc:304] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:06.552402 848 leader_election.cc:343] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.552480 848 leader_election.cc:304] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:06.552778 848 leader_election.cc:343] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:06.553014 848 consensus_peers.cc:487] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5dd41d42331494e9c1b021816d0287e. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:06.553028 710 leader_election.cc:304] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:06.553298 2224 raft_consensus.cc:2798] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:06.553359 2224 raft_consensus.cc:491] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:06.553411 2224 raft_consensus.cc:3054] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.553416 2083 raft_consensus.cc:2743] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.553551 2083 raft_consensus.cc:2743] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.555821 2224 raft_consensus.cc:513] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.556497 2224 leader_election.cc:290] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.561843 1168 tablet_bootstrap.cc:492] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:06.562587 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_519ee6346f5a476f875b3d4b5e249b0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:06.563496 1164 tablet_bootstrap.cc:492] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:06.565212 1164 tablet_bootstrap.cc:654] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.579340 1166 tablet_replica.cc:331] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:06.579617 1166 raft_consensus.cc:2238] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.580183 1166 raft_consensus.cc:2267] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.580799 1166 ts_tablet_manager.cc:1905] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:06.583662 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 36514c5399924e98b3bc38004a33b919 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36514c5399924e98b3bc38004a33b919 already in progress: creating tablet
I20250115 14:52:06.583734 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d4b4b4d0c8e644acb0eaa155e8c59606"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.583911 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 115431us
W20250115 14:52:06.584177 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e9febafb7e354e289cd3f864311f14cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9febafb7e354e289cd3f864311f14cb already in progress: creating tablet
W20250115 14:52:06.583681 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6f2bc62621d402cacfcbe2eb74173f9 consensus state has no leader
I20250115 14:52:06.584367 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet dff155a5c2b0479a90e74993de6f402d (table default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]) successfully deleted
W20250115 14:52:06.584725 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6ce498ffd3f04c5d8eb642739a838903 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ce498ffd3f04c5d8eb642739a838903 already in progress: creating tablet
I20250115 14:52:06.584789 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6472f60e6915482395c4388061880777"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:06.584939 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 5b85d63974ab418eb1921dd605fbf2b0 (table default.Table_854a113be33b4b698a50ae5bd10a2fff [id=f1ce643dc59040148201fdd0c049fb20]) successfully deleted
W20250115 14:52:06.585271 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6472f60e6915482395c4388061880777 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.586046 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 99449d50c2dd46159a69dc0af1de7a87 (DEFAULT_TABLE table=default.Table_201c8e612d7d4a85a6d4d4b0a5095254 [id=b74194f7d2f547cdad4e1e59ddec89d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.586503 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99449d50c2dd46159a69dc0af1de7a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.586589 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d849c4e268a54191ae2632884f61edc0 (DEFAULT_TABLE table=default.Table_c4f5e92e6f994cb0ad035e3cef60b3ce [id=9ed3557619724e11b7875953397cee2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.586948 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d849c4e268a54191ae2632884f61edc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:06.588826 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_519ee6346f5a476f875b3d4b5e249b0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.589834 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 580742eb5dac4dba9995809890e0e315
I20250115 14:52:06.590551 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2814144b33794c15b9386d76197e2e95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.590992 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2814144b33794c15b9386d76197e2e95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.591879 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ba33eb3f5a40450db5cf115fbe8a17e7
I20250115 14:52:06.592478 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f257047f59814e1a8458779252a782b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.592912 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f257047f59814e1a8458779252a782b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.593724 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ded663ac1197454bbde3c4c2d526002d
I20250115 14:52:06.594267 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5dd9ae0549364620b988ae28f376f552" } new_table_name: "default.Table_0f45c2d986f34bb7ae3fcd712eb1548f" modify_external_catalogs: true
W20250115 14:52:06.595216 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7558b4d7453453f83facb0fc7e371e8 consensus state has no leader
I20250115 14:52:06.595423 455 master_service.cc:946] ReplaceTablet: received request to replace tablet b803d8e103a5423f85c75c9006809b17 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:06.596251 450 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:06.596901 450 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet efb8b001dd66456d9e49e9e8a6848ccf (table default.Table_e8f8acbb30db4f019ebeadda783b434a [id=877cb5d7df394006822f9a0e8a133dcc]). Expected version 1 got 0
W20250115 14:52:06.601186 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4b4b4d0c8e644acb0eaa155e8c59606 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.602874 447 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet fbf94077e0e343b8938b081102dfd2af (table default.Table_bbd25ad975704fbcae339bf950fd1052 [id=00a89ba1b93f4132aae8fe6e748df77e]). Expected version 2 got 0
I20250115 14:52:06.565002 2200 consensus_queue.cc:237] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:06.589468 848 leader_election.cc:343] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.594329 1167 rpcz_store.cc:276] 0115 14:52:04.583403 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.584622 (+ 1219us) service_pool.cc:225] Handling call
0115 14:52:06.393075 (+1808453us) ts_tablet_manager.cc:1145] Deleting tablet 105b6148ae944e5c906e07d987177cc5
0115 14:52:06.393083 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.393149 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808280}
I20250115 14:52:06.605719 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 687) took 1898 ms. Trace:
I20250115 14:52:06.605801 1167 rpcz_store.cc:276] 0115 14:52:04.707501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.707547 (+ 46us) service_pool.cc:225] Handling call
0115 14:52:06.605601 (+1898054us) ts_tablet_manager.cc:1145] Deleting tablet ca5cc55b50ee4953800f10efe0ffb546
0115 14:52:06.605611 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.605702 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1894651}
I20250115 14:52:06.606096 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 686) took 1909 ms. Trace:
I20250115 14:52:06.606160 1167 rpcz_store.cc:276] 0115 14:52:04.697030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.716714 (+ 19684us) service_pool.cc:225] Handling call
0115 14:52:06.606001 (+1889287us) ts_tablet_manager.cc:1145] Deleting tablet 48b15de9bf8943cdb9e5abd943484993
0115 14:52:06.606008 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.606084 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1888312}
I20250115 14:52:06.606370 1167 tablet_replica.cc:331] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:06.606529 1167 raft_consensus.cc:2238] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.606645 1167 raft_consensus.cc:2267] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.606793 1168 ts_tablet_manager.cc:1397] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.280s user 0.002s sys 0.015s
I20250115 14:52:06.607101 1167 ts_tablet_manager.cc:1905] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.605720 886 tablet_service.cc:1467] Processing CreateTablet for tablet 99449d50c2dd46159a69dc0af1de7a87 (DEFAULT_TABLE table=default.Table_201c8e612d7d4a85a6d4d4b0a5095254 [id=b74194f7d2f547cdad4e1e59ddec89d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.608881 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99449d50c2dd46159a69dc0af1de7a87. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:06.610118 847 consensus_peers.cc:487] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 009ddfe9dfe14ba2b067f907a9d13538. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:06.610143 893 tablet_service.cc:1467] Processing CreateTablet for tablet d849c4e268a54191ae2632884f61edc0 (DEFAULT_TABLE table=default.Table_c4f5e92e6f994cb0ad035e3cef60b3ce [id=9ed3557619724e11b7875953397cee2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.610287 1168 raft_consensus.cc:357] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.610455 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d849c4e268a54191ae2632884f61edc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.610481 1168 raft_consensus.cc:383] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.610531 1168 raft_consensus.cc:738] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.610757 1168 consensus_queue.cc:260] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.611352 1168 ts_tablet_manager.cc:1428] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250115 14:52:06.611588 1168 tablet_bootstrap.cc:492] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:06.619871 1167 ts_tablet_manager.cc:1918] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.620080 2224 raft_consensus.cc:491] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.620379 2224 raft_consensus.cc:513] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.621030 2224 leader_election.cc:290] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.621230 1167 log.cc:1198] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/010e6535b6bc44f391e6f2b6f1686532
I20250115 14:52:06.621547 1167 ts_tablet_manager.cc:1939] T 010e6535b6bc44f391e6f2b6f1686532 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:06.621848 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 688) took 1895 ms. Trace:
W20250115 14:52:06.620465 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 67456.4us
I20250115 14:52:06.624111 1167 rpcz_store.cc:276] 0115 14:52:04.726093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.726359 (+ 266us) service_pool.cc:225] Handling call
0115 14:52:06.606330 (+1879971us) ts_tablet_manager.cc:1145] Deleting tablet 010e6535b6bc44f391e6f2b6f1686532
0115 14:52:06.606338 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.614961 (+ 8623us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.619801 (+ 4840us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.621837 (+ 2036us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879749}
I20250115 14:52:06.624658 1167 tablet_replica.cc:331] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
W20250115 14:52:06.624786 848 consensus_peers.cc:487] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:06.624935 1167 raft_consensus.cc:2238] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.625115 1167 raft_consensus.cc:2267] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.625514 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f793828f2ca49639c57849afc61ce9e" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:06.625599 771 raft_consensus.cc:3054] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.625633 1167 ts_tablet_manager.cc:1905] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.626159 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f793828f2ca49639c57849afc61ce9e" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:06.627310 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 009ddfe9dfe14ba2b067f907a9d13538 (table default.Table_96eab6cd20b842138b5acc7c8d0ac5d6 [id=816d14738e72406a88b9eed42d72be73]) successfully deleted
I20250115 14:52:06.628541 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 580742eb5dac4dba9995809890e0e315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.628767 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ba33eb3f5a40450db5cf115fbe8a17e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:06.628885 710 leader_election.cc:343] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 010e6535b6bc44f391e6f2b6f1686532
I20250115 14:52:06.628923 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ded663ac1197454bbde3c4c2d526002d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:06.629026 710 leader_election.cc:343] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 010e6535b6bc44f391e6f2b6f1686532
I20250115 14:52:06.629099 710 leader_election.cc:304] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:52:06.629237 710 raft_consensus.cc:2681] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:52:06.629483 710 leader_election.cc:304] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:06.629814 2224 raft_consensus.cc:2743] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.634279 1167 ts_tablet_manager.cc:1918] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.634505 2186 raft_consensus.cc:491] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.634640 2186 raft_consensus.cc:513] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.635383 2186 leader_election.cc:290] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.638309 772 raft_consensus.cc:3054] T 1f793828f2ca49639c57849afc61ce9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.642514 1166 ts_tablet_manager.cc:1918] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.652734 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 026d05f8fb584da39000d83546b09e63 (DEFAULT_TABLE table=default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.653170 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026d05f8fb584da39000d83546b09e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.636411 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb8b001dd66456d9e49e9e8a6848ccf" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:06.658237 1167 log.cc:1198] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/caa1827366a34f039fd12deb855f07b6
I20250115 14:52:06.658537 1167 ts_tablet_manager.cc:1939] T caa1827366a34f039fd12deb855f07b6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:06.658812 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 690) took 1838 ms. Trace:
I20250115 14:52:06.658885 1167 rpcz_store.cc:276] 0115 14:52:04.820538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.820591 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:06.624616 (+1804025us) ts_tablet_manager.cc:1145] Deleting tablet caa1827366a34f039fd12deb855f07b6
0115 14:52:06.624625 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.628206 (+ 3581us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.634207 (+ 6001us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.658796 (+ 24589us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1803836}
I20250115 14:52:06.659233 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 691) took 1796 ms. Trace:
I20250115 14:52:06.659292 1167 rpcz_store.cc:276] 0115 14:52:04.862489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.862697 (+ 208us) service_pool.cc:225] Handling call
0115 14:52:06.659126 (+1796429us) ts_tablet_manager.cc:1145] Deleting tablet 83415f1c07ad495cabd287bdcd54dad4
0115 14:52:06.659134 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.659221 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1796260}
I20250115 14:52:06.659523 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 692) took 1794 ms. Trace:
I20250115 14:52:06.659571 1167 rpcz_store.cc:276] 0115 14:52:04.864631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.864863 (+ 232us) service_pool.cc:225] Handling call
0115 14:52:06.659456 (+1794593us) ts_tablet_manager.cc:1145] Deleting tablet a48a0af9422f43cf86531665c0fa1aee
0115 14:52:06.659462 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.659513 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794442}
I20250115 14:52:06.637498 2200 consensus_queue.cc:1035] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:06.659790 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 693) took 1788 ms. Trace:
I20250115 14:52:06.659843 1167 rpcz_store.cc:276] 0115 14:52:04.871387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.871432 (+ 45us) service_pool.cc:225] Handling call
0115 14:52:06.659719 (+1788287us) ts_tablet_manager.cc:1145] Deleting tablet cc9ce226d3ba44d6908257bbd0df3a12
0115 14:52:06.659726 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.659778 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1788146}
I20250115 14:52:06.642913 2083 consensus_queue.cc:1035] T 1f793828f2ca49639c57849afc61ce9e P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:06.660080 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 694) took 1784 ms. Trace:
I20250115 14:52:06.660130 1167 rpcz_store.cc:276] 0115 14:52:04.875363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.875405 (+ 42us) service_pool.cc:225] Handling call
0115 14:52:06.659999 (+1784594us) ts_tablet_manager.cc:1145] Deleting tablet c37f4c2ab9d248a1af817cfa2c2e2ff0
0115 14:52:06.660006 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.660069 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1784456}
I20250115 14:52:06.660629 1166 log.cc:1198] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/010e6535b6bc44f391e6f2b6f1686532
W20250115 14:52:06.661044 983 leader_election.cc:343] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.651909 886 tablet_service.cc:1467] Processing CreateTablet for tablet 026d05f8fb584da39000d83546b09e63 (DEFAULT_TABLE table=default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.663623 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026d05f8fb584da39000d83546b09e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.665844 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 696) took 1739 ms. Trace:
I20250115 14:52:06.665910 1167 rpcz_store.cc:276] 0115 14:52:04.926639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:04.926688 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:06.660283 (+1733595us) ts_tablet_manager.cc:1145] Deleting tablet fa57c1fa96594a40b2551ad503127e4f
0115 14:52:06.665730 (+ 5447us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.665827 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1733433}
I20250115 14:52:06.650509 1164 tablet_bootstrap.cc:492] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:06.666380 1164 ts_tablet_manager.cc:1397] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.103s user 0.003s sys 0.000s
I20250115 14:52:06.666411 1167 tablet_replica.cc:331] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:06.666579 1167 raft_consensus.cc:2238] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.666703 1167 raft_consensus.cc:2267] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.667176 1167 ts_tablet_manager.cc:1905] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.667058 1164 raft_consensus.cc:357] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.667295 1164 raft_consensus.cc:383] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.667346 1164 raft_consensus.cc:738] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.667572 1164 consensus_queue.cc:260] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.675419 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_bbd25ad975704fbcae339bf950fd1052 [id=00a89ba1b93f4132aae8fe6e748df77e] alter complete (version 2)
W20250115 14:52:06.675442 848 consensus_peers.cc:487] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:06.676537 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c16ddd6216634793b8438ac0421e8cb5" } modify_external_catalogs: true
I20250115 14:52:06.676726 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c16ddd6216634793b8438ac0421e8cb5" } modify_external_catalogs: true
I20250115 14:52:06.677573 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1e83e61b373d4455b007c7214a0dba53" } modify_external_catalogs: true
I20250115 14:52:06.677752 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1e83e61b373d4455b007c7214a0dba53" } modify_external_catalogs: true
I20250115 14:52:06.678097 1162 tablet_bootstrap.cc:492] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:06.678215 1162 ts_tablet_manager.cc:1397] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.251s user 0.006s sys 0.000s
I20250115 14:52:06.678383 452 master_service.cc:946] ReplaceTablet: received request to replace tablet fbf5da59db79418498db09ec8ea3ac0a from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:06.678845 1162 raft_consensus.cc:357] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.679011 1162 raft_consensus.cc:383] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.679066 1162 raft_consensus.cc:738] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.678958 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d60ab02c55f94f3590d4b43fb3c8fa0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:06.679301 1162 consensus_queue.cc:260] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:06.679473 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d60ab02c55f94f3590d4b43fb3c8fa0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.679767 1162 ts_tablet_manager.cc:1428] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250115 14:52:06.679982 1162 tablet_bootstrap.cc:492] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.680366 2241 raft_consensus.cc:491] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.680474 2241 raft_consensus.cc:513] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.681077 2241 leader_election.cc:290] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.681597 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4341977f363d40eeac71b7d33d1b43ac" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.681914 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4341977f363d40eeac71b7d33d1b43ac" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:06.682041 984 leader_election.cc:343] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.682410 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ec9e96994cca43569c5976b968296550" } new_table_name: "default.Table_52e9521f04f342acb2bad7d736f8847e" modify_external_catalogs: true
W20250115 14:52:06.682564 983 leader_election.cc:343] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.682657 983 leader_election.cc:304] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.682993 2241 raft_consensus.cc:2743] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:06.684260 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78a2d39d0296468db48b3eec374244ca consensus state has no leader
I20250115 14:52:06.684639 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ab3fe6097891488eb31670fe723ef7d6" } new_table_name: "default.Table_1f9e67f6de614b0ba6cd1403e8c3ef5b" modify_external_catalogs: true
I20250115 14:52:06.685824 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b803d8e103a5423f85c75c9006809b17
I20250115 14:52:06.686272 455 catalog_manager.cc:6596] ReplaceTablet: tablet b803d8e103a5423f85c75c9006809b17 deleted and replaced by tablet 6437873733584722979c1e9ae8b0730b
I20250115 14:52:06.686615 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5ce3a92ee6154dee8af40b3ac625550c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.687112 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ce3a92ee6154dee8af40b3ac625550c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.692744 1013 tablet_service.cc:1467] Processing CreateTablet for tablet eba5819df2944a84a636c6934b18510d (DEFAULT_TABLE table=default.Table_352831c0b6b04ca6825725b7825ecb12 [id=39c963a5bc8f4a3b9ff5780bb75e33e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.693208 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eba5819df2944a84a636c6934b18510d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.694293 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d4ae47e4d4274e66adc1753796b0ae1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.694721 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4ae47e4d4274e66adc1753796b0ae1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.701649 886 tablet_service.cc:1467] Processing CreateTablet for tablet 36749313f40648ac8cec4ac7f3d97cf8 (DEFAULT_TABLE table=default.Table_7f8c179068074379b0e5baec32f2699b [id=b1b61ad1a148420585b4d68b675f2055]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.701925 2225 raft_consensus.cc:491] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 73c86158cc2641f182ea50a06634592d)
I20250115 14:52:06.702077 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36749313f40648ac8cec4ac7f3d97cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.702080 2225 raft_consensus.cc:513] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.702745 2225 leader_election.cc:290] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.703689 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84bc24a71e431cb47b93f2b0dcaa90" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:06.704948 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df84bc24a71e431cb47b93f2b0dcaa90" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:52:06.705322 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a739d36ef9d54c089667ff32de875af6 consensus state has no leader
I20250115 14:52:06.705601 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 6942d102069d4c33b23fa2c0b29f94a6 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:06.705471 710 leader_election.cc:304] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:06.705830 2225 raft_consensus.cc:2743] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 14:52:06.706177 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1a17daf7f1be4dcd938b8a2e2f7156d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.706655 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a17daf7f1be4dcd938b8a2e2f7156d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:06.707906 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96369c35ca0d4a62b0676cac3dc4a813 consensus state has no leader
I20250115 14:52:06.708369 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 36749313f40648ac8cec4ac7f3d97cf8 (DEFAULT_TABLE table=default.Table_7f8c179068074379b0e5baec32f2699b [id=b1b61ad1a148420585b4d68b675f2055]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.708850 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36749313f40648ac8cec4ac7f3d97cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.713299 752 tablet_service.cc:1514] Processing DeleteTablet for tablet b803d8e103a5423f85c75c9006809b17 with delete_type TABLET_DATA_DELETED (replaced by tablet 6437873733584722979c1e9ae8b0730b) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.715340 1167 ts_tablet_manager.cc:1918] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.717536 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 79736602ea2646b39708ca9c6834e847 (DEFAULT_TABLE table=default.Table_1e83e61b373d4455b007c7214a0dba53 [id=92d264b71c3a4716b8d930d7b8677f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.717906 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79736602ea2646b39708ca9c6834e847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.717943 446 catalog_manager.cc:5526] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:06.718353 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1]). Expected version 3 got 0
I20250115 14:52:06.718869 446 catalog_manager.cc:5526] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:06.719259 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:06.716888 882 tablet_service.cc:1467] Processing CreateTablet for tablet 79736602ea2646b39708ca9c6834e847 (DEFAULT_TABLE table=default.Table_1e83e61b373d4455b007c7214a0dba53 [id=92d264b71c3a4716b8d930d7b8677f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.720584 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79736602ea2646b39708ca9c6834e847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.720525 893 tablet_service.cc:1467] Processing CreateTablet for tablet eba5819df2944a84a636c6934b18510d (DEFAULT_TABLE table=default.Table_352831c0b6b04ca6825725b7825ecb12 [id=39c963a5bc8f4a3b9ff5780bb75e33e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.724566 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eba5819df2944a84a636c6934b18510d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.725593 1168 tablet_bootstrap.cc:654] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.726104 746 tablet_service.cc:1467] Processing CreateTablet for tablet ad417f06081d41248dff6f1eed175f29 (DEFAULT_TABLE table=default.Table_f047206807ce42eb89594d08a8209682 [id=083c86d9313c40f590d6840429fe1624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.726441 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad417f06081d41248dff6f1eed175f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.726392 755 tablet_service.cc:1467] Processing CreateTablet for tablet b579be5c683e4fd8a5d5484fdb89250c (DEFAULT_TABLE table=default.Table_2179cfd883084fe7864e4cf7bee57581 [id=caea68b0c11f4acebe8b53aea12711df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.726657 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b579be5c683e4fd8a5d5484fdb89250c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.726903 748 tablet_service.cc:1467] Processing CreateTablet for tablet b4248afe06a5400ca0ce38549f82b400 (DEFAULT_TABLE table=default.Table_572841278ecd4e1d9c55e42ea357b3c1 [id=6547c744e1354980ac24674b78afd741]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.726981 879 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad3cda1056349a789a1751c5d386d0c (DEFAULT_TABLE table=default.Table_b28114e5556947c4aaba883427f90c2a [id=2d52564d7100489c803c031d747f6f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.727180 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4248afe06a5400ca0ce38549f82b400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.727289 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad3cda1056349a789a1751c5d386d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.727263 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad3cda1056349a789a1751c5d386d0c (DEFAULT_TABLE table=default.Table_b28114e5556947c4aaba883427f90c2a [id=2d52564d7100489c803c031d747f6f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.727623 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ad3cda1056349a789a1751c5d386d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.729348 752 tablet_service.cc:1467] Processing CreateTablet for tablet a6270d3c4ee94679a92f8bbfd1aa5d95 (DEFAULT_TABLE table=default.Table_e37f4580bc3c4d5cacb53a430d3d5605 [id=6087ca0bad3147b9ace51f422882725c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.729992 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6270d3c4ee94679a92f8bbfd1aa5d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.731106 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_092faa925c9f45838cbc3a14b0fc60dd [id=5d05be5a21a44fe0b75c284bf3c83625] alter complete (version 2)
I20250115 14:52:06.731428 744 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7b686a6b4043b78a9a59292bcc34ba (DEFAULT_TABLE table=default.Table_3e03f479683449e197a5150099b49504 [id=6801ef0ff2474c348248d37b86192981]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.731694 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7b686a6b4043b78a9a59292bcc34ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.732364 751 tablet_service.cc:1467] Processing CreateTablet for tablet 11aeb6c73b4d42058736ffce551e311c (DEFAULT_TABLE table=default.Table_69ed5f68465549b38e7e27bf0d75eb2e [id=f08c5a8508224d27a45438fe8bd9390e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.732626 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11aeb6c73b4d42058736ffce551e311c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.732844 875 tablet_service.cc:1467] Processing CreateTablet for tablet 9c35882d2e5c464c9cfad496ba9d6af9 (DEFAULT_TABLE table=default.Table_958850c3f0a54c49a8dd42b180068897 [id=8242d7a08fcd4d4e836f6b70b1bacdc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.732977 740 tablet_service.cc:1467] Processing CreateTablet for tablet ff771f5acdcf4fcb9e099aede920f0a1 (DEFAULT_TABLE table=default.Table_b1ed6d26296a4f9799945452887c3c1c [id=0a128de980814ba281cf1bc3b6e26fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.733131 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c35882d2e5c464c9cfad496ba9d6af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.733455 891 tablet_service.cc:1467] Processing CreateTablet for tablet bc1440e3cd444cf8b3365a0af8de8958 (DEFAULT_TABLE table=default.Table_6f0b6c724c1c43e7adf538ad7369b7a1 [id=9e8ce51e784e4bdbb3ac78cba10bfedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.733526 1010 tablet_service.cc:1467] Processing CreateTablet for tablet bc1440e3cd444cf8b3365a0af8de8958 (DEFAULT_TABLE table=default.Table_6f0b6c724c1c43e7adf538ad7369b7a1 [id=9e8ce51e784e4bdbb3ac78cba10bfedf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.733721 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc1440e3cd444cf8b3365a0af8de8958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.733827 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc1440e3cd444cf8b3365a0af8de8958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.736558 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 9c35882d2e5c464c9cfad496ba9d6af9 (DEFAULT_TABLE table=default.Table_958850c3f0a54c49a8dd42b180068897 [id=8242d7a08fcd4d4e836f6b70b1bacdc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.736907 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c35882d2e5c464c9cfad496ba9d6af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.741370 741 tablet_service.cc:1467] Processing CreateTablet for tablet 00e6cb19641f49e599275a5a5fc272fb (DEFAULT_TABLE table=default.Table_b02e318b61ed4745bf6bf686cd720090 [id=aaa486d6ed38402a98803f39ad3e5c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.741605 879 tablet_service.cc:1467] Processing CreateTablet for tablet 546fe492182f47b7b008f4719f97e877 (DEFAULT_TABLE table=default.Table_ba297fd1bf8c47dabb7b1dab458cf8df [id=d0926ed2b8324e67bbcf6ae4b4a2154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.741683 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e6cb19641f49e599275a5a5fc272fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.741906 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 546fe492182f47b7b008f4719f97e877. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.742220 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet abccc613e1754839a4c7a8f5c448a852 with delete_type TABLET_DATA_DELETED (replaced by tablet 11aeb6c73b4d42058736ffce551e311c) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:06.742489 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff771f5acdcf4fcb9e099aede920f0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.742624 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0d97fa57484b9d8cb879fedda61612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:06.743240 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet c14836b97c3247b39ff9072c12e642e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:06.744138 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet f60e68b411364ada9b89879012d862ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:06.744796 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8efcf1a2db44469cd8dda0b59c34ba) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:06.745234 749 tablet_service.cc:1467] Processing CreateTablet for tablet d360add9d39a48caa7b90f7d786f7bc1 (DEFAULT_TABLE table=default.Table_76b7b4a0c17041068bfb2be07cf21267 [id=df847f2fc5394b74832182b18e3e1701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.745535 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d360add9d39a48caa7b90f7d786f7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.746739 880 tablet_service.cc:1514] Processing DeleteTablet for tablet abccc613e1754839a4c7a8f5c448a852 with delete_type TABLET_DATA_DELETED (replaced by tablet 11aeb6c73b4d42058736ffce551e311c) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:06.746905 876 tablet_service.cc:1514] Processing DeleteTablet for tablet c14836b97c3247b39ff9072c12e642e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:06.747210 885 tablet_service.cc:1514] Processing DeleteTablet for tablet f60e68b411364ada9b89879012d862ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:06.751565 1751 ts_tablet_manager.cc:1918] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.751722 1751 log.cc:1198] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/8e6cc08065494d9e85eeff5044b44ba6
I20250115 14:52:06.752276 1751 ts_tablet_manager.cc:1939] T 8e6cc08065494d9e85eeff5044b44ba6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:06.753819 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 765) took 1172 ms. Trace:
I20250115 14:52:06.754841 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 546fe492182f47b7b008f4719f97e877 (DEFAULT_TABLE table=default.Table_ba297fd1bf8c47dabb7b1dab458cf8df [id=d0926ed2b8324e67bbcf6ae4b4a2154d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.755298 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 546fe492182f47b7b008f4719f97e877. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.756196 1164 ts_tablet_manager.cc:1428] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.090s user 0.001s sys 0.000s
I20250115 14:52:06.756482 1164 tablet_bootstrap.cc:492] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:06.758450 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 104282a09c024bbaafeb5f28f91dadcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:06.647747 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb8b001dd66456d9e49e9e8a6848ccf" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:06.759274 984 leader_election.cc:343] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.760337 984 leader_election.cc:304] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.760607 2241 raft_consensus.cc:2743] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.762061 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 79736602ea2646b39708ca9c6834e847
I20250115 14:52:06.762871 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 40eae66b35974524992b2ef943273291
I20250115 14:52:06.762879 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 79736602ea2646b39708ca9c6834e847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.763520 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 40eae66b35974524992b2ef943273291 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.764918 882 tablet_service.cc:1467] Processing CreateTablet for tablet f3be5fb895424a3498c22d0108034ad1 (DEFAULT_TABLE table=default.Table_38055c6c2fc748eb9fcefff19f2e92e9 [id=63588086fdf344819a066c04501cab6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.765304 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3be5fb895424a3498c22d0108034ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.768991 1166 ts_tablet_manager.cc:1939] T 010e6535b6bc44f391e6f2b6f1686532 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:06.770362 1167 log.cc:1198] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/5bc975b044dd439dbd1f8f9a82cce751
I20250115 14:52:06.770628 1167 ts_tablet_manager.cc:1939] T 5bc975b044dd439dbd1f8f9a82cce751 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:06.770895 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 697) took 1714 ms. Trace:
I20250115 14:52:06.770972 1167 rpcz_store.cc:276] 0115 14:52:05.056660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.056746 (+ 86us) service_pool.cc:225] Handling call
0115 14:52:06.666372 (+1609626us) ts_tablet_manager.cc:1145] Deleting tablet 5bc975b044dd439dbd1f8f9a82cce751
0115 14:52:06.666380 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.697352 (+ 30972us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.715238 (+ 17886us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.770882 (+ 55644us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1608660}
I20250115 14:52:06.771376 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 699) took 1710 ms. Trace:
I20250115 14:52:06.771456 1167 rpcz_store.cc:276] 0115 14:52:05.060466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.060534 (+ 68us) service_pool.cc:225] Handling call
0115 14:52:06.771266 (+1710732us) ts_tablet_manager.cc:1145] Deleting tablet 0a2bcfc5c5714a3eac85b6a8beafbf59
0115 14:52:06.771275 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.771361 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1709972}
I20250115 14:52:06.771759 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 703) took 1632 ms. Trace:
I20250115 14:52:06.771845 1167 rpcz_store.cc:276] 0115 14:52:05.139474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.139771 (+ 297us) service_pool.cc:225] Handling call
0115 14:52:06.771668 (+1631897us) ts_tablet_manager.cc:1145] Deleting tablet fb81e8fcc38a41e69cfb0e34dbaaac84
0115 14:52:06.771676 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.771746 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1631738}
I20250115 14:52:06.772135 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 704) took 1632 ms. Trace:
I20250115 14:52:06.772208 1167 rpcz_store.cc:276] 0115 14:52:05.139555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.139959 (+ 404us) service_pool.cc:225] Handling call
0115 14:52:06.772045 (+1632086us) ts_tablet_manager.cc:1145] Deleting tablet d1259256b7744e739581125611abbe67
0115 14:52:06.772054 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.772122 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1631975}
I20250115 14:52:06.772403 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 010e6535b6bc44f391e6f2b6f1686532 (table default.Table_c5f747dc2f6b4cffb7b791ea6456c5b3 [id=13d88a401bed452688d5bb6f4dcfd70b]) successfully deleted
I20250115 14:52:06.772505 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 705) took 1632 ms. Trace:
I20250115 14:52:06.772558 1167 rpcz_store.cc:276] 0115 14:52:05.139645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.140089 (+ 444us) service_pool.cc:225] Handling call
0115 14:52:06.772420 (+1632331us) ts_tablet_manager.cc:1145] Deleting tablet 65a6f83730f74d13823bd15d93538df7
0115 14:52:06.772428 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.772492 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1632243}
W20250115 14:52:06.772670 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c14836b97c3247b39ff9072c12e642e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c14836b97c3247b39ff9072c12e642e0 already in progress: creating tablet
I20250115 14:52:06.772828 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 706) took 1633 ms. Trace:
W20250115 14:52:06.772892 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet abccc613e1754839a4c7a8f5c448a852 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abccc613e1754839a4c7a8f5c448a852 already in progress: creating tablet
I20250115 14:52:06.772888 1167 rpcz_store.cc:276] 0115 14:52:05.139726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.140190 (+ 464us) service_pool.cc:225] Handling call
0115 14:52:06.772735 (+1632545us) ts_tablet_manager.cc:1145] Deleting tablet 57aecd22f97c41618f90d14c2f53a7c9
0115 14:52:06.772743 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.772815 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1632438}
W20250115 14:52:06.773092 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f60e68b411364ada9b89879012d862ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f60e68b411364ada9b89879012d862ea already in progress: creating tablet
I20250115 14:52:06.773144 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 707) took 1613 ms. Trace:
I20250115 14:52:06.773195 1167 rpcz_store.cc:276] 0115 14:52:05.159215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.159264 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:06.773061 (+1613797us) ts_tablet_manager.cc:1145] Deleting tablet be8e12ccb8d14c1c9258dc3f81d1527d
0115 14:52:06.773068 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.773130 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1613645}
W20250115 14:52:06.773290 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 580742eb5dac4dba9995809890e0e315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 580742eb5dac4dba9995809890e0e315 already in progress: creating tablet
W20250115 14:52:06.773487 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ba33eb3f5a40450db5cf115fbe8a17e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba33eb3f5a40450db5cf115fbe8a17e7 already in progress: creating tablet
I20250115 14:52:06.773471 1028 tablet_service.cc:1467] Processing CreateTablet for tablet f3be5fb895424a3498c22d0108034ad1 (DEFAULT_TABLE table=default.Table_38055c6c2fc748eb9fcefff19f2e92e9 [id=63588086fdf344819a066c04501cab6c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:06.773691 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ded663ac1197454bbde3c4c2d526002d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ded663ac1197454bbde3c4c2d526002d already in progress: creating tablet
I20250115 14:52:06.773828 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3be5fb895424a3498c22d0108034ad1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:06.773844 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b803d8e103a5423f85c75c9006809b17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b803d8e103a5423f85c75c9006809b17 already in progress: creating tablet
W20250115 14:52:06.774049 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 79736602ea2646b39708ca9c6834e847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79736602ea2646b39708ca9c6834e847 already in progress: creating tablet
I20250115 14:52:06.774567 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e93ba6767713429194e5e07a2b1afc59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.774837 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 40eae66b35974524992b2ef943273291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40eae66b35974524992b2ef943273291 already in progress: creating tablet
W20250115 14:52:06.775161 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e93ba6767713429194e5e07a2b1afc59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.776780 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 792c68ba824640c1b653ad0bfbe2de02 (DEFAULT_TABLE table=default.Table_21e00049c7364adb9a87e33cd7a04957 [id=94f032d2cb6c42f9aea0538a13bed768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.777086 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 792c68ba824640c1b653ad0bfbe2de02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.781025 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 9449e1d09c3148ee86074a57c8e22240 (DEFAULT_TABLE table=default.Table_78ff869a9d0c483a87b4d43f8bcac7c9 [id=629bac9d176a4083ab706748a9a7252e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.781313 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9449e1d09c3148ee86074a57c8e22240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.784567 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9449e1d09c3148ee86074a57c8e22240 (DEFAULT_TABLE table=default.Table_78ff869a9d0c483a87b4d43f8bcac7c9 [id=629bac9d176a4083ab706748a9a7252e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.784976 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9449e1d09c3148ee86074a57c8e22240. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.786495 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f3220f35253d4aa087fd1d48bd8dc8b3" } new_table_name: "default.Table_57f588c93610423484d30d343513c135" modify_external_catalogs: true
I20250115 14:52:06.787359 1162 tablet_bootstrap.cc:654] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.793265 885 tablet_service.cc:1467] Processing CreateTablet for tablet 75ccc1f31c6a4d2cba65324b7a71c2ce (DEFAULT_TABLE table=default.Table_55bce736d0234efa9eeee2a1745b955a [id=67af6a5e607747ac8e6831dc2051b1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.793452 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 75ccc1f31c6a4d2cba65324b7a71c2ce (DEFAULT_TABLE table=default.Table_55bce736d0234efa9eeee2a1745b955a [id=67af6a5e607747ac8e6831dc2051b1ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.793608 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ccc1f31c6a4d2cba65324b7a71c2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.793747 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75ccc1f31c6a4d2cba65324b7a71c2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.793761 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3df4c6d310d7465ca769a5b3a13b8f59 (DEFAULT_TABLE table=default.Table_f459a15fa0234ba9b5722b7bbff3b6cb [id=a5a9ecce138e4b7aa6a792e00d79d997]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.794150 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df4c6d310d7465ca769a5b3a13b8f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.797185 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2179cfd883084fe7864e4cf7bee57581" } modify_external_catalogs: true
I20250115 14:52:06.797363 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2179cfd883084fe7864e4cf7bee57581" } modify_external_catalogs: true
I20250115 14:52:06.798540 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_341db1cd78b3454789f331ecd640e902" } modify_external_catalogs: true
I20250115 14:52:06.798702 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_341db1cd78b3454789f331ecd640e902" } modify_external_catalogs: true
I20250115 14:52:06.800391 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8f606b5f353c41be81616521eb8423f3" } modify_external_catalogs: true
I20250115 14:52:06.800541 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8f606b5f353c41be81616521eb8423f3" } modify_external_catalogs: true
I20250115 14:52:06.801738 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9da42aab96b54c6f84d977c95677d7e3" } new_table_name: "default.Table_ab770caafef447218be7ae6edcd11e57" modify_external_catalogs: true
I20250115 14:52:06.801862 1168 tablet_bootstrap.cc:492] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:06.801987 1168 ts_tablet_manager.cc:1397] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.190s user 0.002s sys 0.002s
I20250115 14:52:06.802574 1168 raft_consensus.cc:357] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.803277 1168 raft_consensus.cc:383] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.803550 1168 raft_consensus.cc:738] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.804054 1168 consensus_queue.cc:260] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.805122 1168 ts_tablet_manager.cc:1428] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:06.806202 1168 tablet_bootstrap.cc:492] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:06.808367 1168 tablet_bootstrap.cc:654] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.808799 1164 tablet_bootstrap.cc:654] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.811125 2234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:52:06.811491 2234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:52:06.813436 2059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250115 14:52:06.816521 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 711) took 1615 ms. Trace:
I20250115 14:52:06.816651 1167 rpcz_store.cc:276] 0115 14:52:05.201118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.201165 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:06.816403 (+1615238us) ts_tablet_manager.cc:1145] Deleting tablet 567365040b6d4d6eba04112c8d4c3dfb
0115 14:52:06.816412 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.816504 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1615078}
I20250115 14:52:06.817031 1167 tablet_replica.cc:331] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:06.817245 1167 raft_consensus.cc:2238] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:06.817538 1167 pending_rounds.cc:62] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:06.817641 1167 pending_rounds.cc:69] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558363696693248 op_type: NO_OP noop_request { }
I20250115 14:52:06.817806 1167 raft_consensus.cc:2883] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:06.817939 1167 raft_consensus.cc:2267] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.818049 2083 consensus_queue.cc:1035] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:06.818468 1167 ts_tablet_manager.cc:1905] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.822902 885 tablet_service.cc:1467] Processing CreateTablet for tablet 93e377a771a0427db8fe8135cdaa1ff3 (DEFAULT_TABLE table=default.Table_679ebe5f3b29426082b31ce8a77da689 [id=da2f0bc60826498fabaf63c5ece311ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.823261 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e377a771a0427db8fe8135cdaa1ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.823424 876 tablet_service.cc:1467] Processing CreateTablet for tablet f1e3879802464acaa279dfe4fbb634bb (DEFAULT_TABLE table=default.Table_b84df010f86642d4b3585c0e774575bf [id=1d71f4010b0f4682ab1a107c0806129b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.823704 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e3879802464acaa279dfe4fbb634bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.830247 876 tablet_service.cc:1467] Processing CreateTablet for tablet 4e39e13fb8c84ae18bac5e45d576a0ab (DEFAULT_TABLE table=default.Table_81803594c5c741619ae708256092f827 [id=caa6adb7563c41088e5120993c46c54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.830601 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e39e13fb8c84ae18bac5e45d576a0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.831199 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 7b6ea13ef1884d66bd1e7655df3e925d (DEFAULT_TABLE table=default.Table_380c0dd6df914022a741ea83ed384911 [id=190589a22d424365a7f858a214fd0f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.831580 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b6ea13ef1884d66bd1e7655df3e925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.833488 885 tablet_service.cc:1467] Processing CreateTablet for tablet 7b6ea13ef1884d66bd1e7655df3e925d (DEFAULT_TABLE table=default.Table_380c0dd6df914022a741ea83ed384911 [id=190589a22d424365a7f858a214fd0f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.833770 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b6ea13ef1884d66bd1e7655df3e925d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.833894 2241 raft_consensus.cc:491] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.834008 2241 raft_consensus.cc:513] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.834578 2241 leader_election.cc:290] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.835520 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a8000f5474f62aae52c8e88e45276" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:06.836043 983 leader_election.cc:343] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.836105 886 tablet_service.cc:1467] Processing CreateTablet for tablet b126874bae6e4c84bd97f5ff65e0882b (DEFAULT_TABLE table=default.Table_340265c995a649808c582e4c4e6d0b22 [id=bf2a1ee254eb411dbbaccc0a78dfbad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.836391 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b126874bae6e4c84bd97f5ff65e0882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.840394 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 4e39e13fb8c84ae18bac5e45d576a0ab (DEFAULT_TABLE table=default.Table_81803594c5c741619ae708256092f827 [id=caa6adb7563c41088e5120993c46c54f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.840766 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e39e13fb8c84ae18bac5e45d576a0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.845269 1168 tablet_bootstrap.cc:492] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:06.845404 1168 ts_tablet_manager.cc:1397] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.004s
I20250115 14:52:06.846064 1168 raft_consensus.cc:357] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.846303 1168 raft_consensus.cc:383] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.846357 1168 raft_consensus.cc:738] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.846534 1168 consensus_queue.cc:260] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:06.846627 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e663111811b34a578bc344c59be8649e consensus state has no leader
I20250115 14:52:06.846669 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7ff71ded3b074ff0979e2090131f4eaa
I20250115 14:52:06.846997 1168 ts_tablet_manager.cc:1428] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:06.846088 2083 consensus_queue.cc:1035] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250115 14:52:06.849762 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 338706b2d7514f94afd3579da458b4ec
W20250115 14:52:06.851174 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b47502309761449dae3be07ea6363238 consensus state has no leader
I20250115 14:52:06.851711 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b579be5c683e4fd8a5d5484fdb89250c
I20250115 14:52:06.854765 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 338706b2d7514f94afd3579da458b4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.856606 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b579be5c683e4fd8a5d5484fdb89250c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:06.858091 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 338706b2d7514f94afd3579da458b4ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 338706b2d7514f94afd3579da458b4ec already in progress: creating tablet
W20250115 14:52:06.858676 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b579be5c683e4fd8a5d5484fdb89250c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b579be5c683e4fd8a5d5484fdb89250c already in progress: creating tablet
I20250115 14:52:06.859797 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ff71ded3b074ff0979e2090131f4eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:06.860666 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7ff71ded3b074ff0979e2090131f4eaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ff71ded3b074ff0979e2090131f4eaa already in progress: creating tablet
I20250115 14:52:06.863847 752 tablet_service.cc:1514] Processing DeleteTablet for tablet be63bae96c834a8bbbb1174c027935a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:06.864187 1166 tablet_replica.cc:331] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:06.864427 1166 raft_consensus.cc:2238] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.864606 1166 raft_consensus.cc:2267] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.865147 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 064f890830304758a6840a1502e59080 (DEFAULT_TABLE table=default.Table_adcff7acc18b46e79d647e28af423fbd [id=d6f2abb6e8894bf58f3aa4876cc38ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.863801 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a8000f5474f62aae52c8e88e45276" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.865473 1166 ts_tablet_manager.cc:1905] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.865589 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 064f890830304758a6840a1502e59080. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:06.865799 984 leader_election.cc:343] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.865900 984 leader_election.cc:304] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.866248 2241 raft_consensus.cc:2743] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.866623 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1df001f0c00149609e20cf9dd049e8b5 (DEFAULT_TABLE table=default.Table_14622d246cb34c0aa0c8527eebb4319b [id=054f4d9637c94374b720f86b1c96d509]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.866959 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df001f0c00149609e20cf9dd049e8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.872433 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 1df001f0c00149609e20cf9dd049e8b5 (DEFAULT_TABLE table=default.Table_14622d246cb34c0aa0c8527eebb4319b [id=054f4d9637c94374b720f86b1c96d509]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.872852 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df001f0c00149609e20cf9dd049e8b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.876313 2225 raft_consensus.cc:491] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.876438 2225 raft_consensus.cc:513] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.877007 2225 leader_election.cc:290] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.878909 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e5d44b8064abbbf1b6419a1eca1ed" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:06.879637 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 3df4c6d310d7465ca769a5b3a13b8f59 (DEFAULT_TABLE table=default.Table_f459a15fa0234ba9b5722b7bbff3b6cb [id=a5a9ecce138e4b7aa6a792e00d79d997]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.880007 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df4c6d310d7465ca769a5b3a13b8f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.882511 1168 tablet_bootstrap.cc:492] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:06.883306 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f1e5d44b8064abbbf1b6419a1eca1ed" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:06.883530 1046 raft_consensus.cc:2388] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d7ecece2ab49e1be7079216517d13b in current term 1: Already voted for candidate 73c86158cc2641f182ea50a06634592d in this term.
I20250115 14:52:06.884555 1020 tablet_service.cc:1467] Processing CreateTablet for tablet f1e3879802464acaa279dfe4fbb634bb (DEFAULT_TABLE table=default.Table_b84df010f86642d4b3585c0e774575bf [id=1d71f4010b0f4682ab1a107c0806129b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.884668 710 leader_election.cc:304] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:06.884794 1168 tablet_bootstrap.cc:654] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.885196 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e3879802464acaa279dfe4fbb634bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.885685 2225 raft_consensus.cc:3054] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.888476 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa67705596742c6b942f49bca0aa817 (DEFAULT_TABLE table=default.Table_3b9fd4f4e09a40e99f015f099eef8306 [id=254935fbfd31462c95a7762ab90fff86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.888871 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa67705596742c6b942f49bca0aa817. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.890844 452 master_service.cc:946] ReplaceTablet: received request to replace tablet ff9b57b8b9434f55ace5a4f47380df54 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:06.891842 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_69711e6d76ee4d289ccb17b881ec97bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.892374 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69711e6d76ee4d289ccb17b881ec97bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.892521 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 93e377a771a0427db8fe8135cdaa1ff3 (DEFAULT_TABLE table=default.Table_679ebe5f3b29426082b31ce8a77da689 [id=da2f0bc60826498fabaf63c5ece311ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.892827 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93e377a771a0427db8fe8135cdaa1ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.894021 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d1c9cfaca7974878b6bf92f9b8cdbbf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.894516 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1c9cfaca7974878b6bf92f9b8cdbbf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.895303 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1f3c5d983c50484da1dc4417e5698154" } new_table_name: "default.Table_2f8249110b37422b8edaa6c0dab8eb60" modify_external_catalogs: true
I20250115 14:52:06.895648 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_eb0ebfb0281048dbae59684c7b4e0269"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.896046 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb0ebfb0281048dbae59684c7b4e0269 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.897660 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d100b9e08d3045d083f5800a8d651450"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.898095 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d100b9e08d3045d083f5800a8d651450 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.899413 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cffb7afceb4a4e8abebbe51a06b58b7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.899855 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cffb7afceb4a4e8abebbe51a06b58b7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.901542 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet a915e0c0a30940ed8a6233f117e2a560 (table default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]) successfully deleted
I20250115 14:52:06.901769 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 37ad2b7350294375bd30fb4b46906491 (table default.Table_0826262d07e14081b5a0ab88d9d4a20f [id=fc679ffd67e74e6c85318eda3acb09a6]) successfully deleted
I20250115 14:52:06.901844 1162 tablet_bootstrap.cc:492] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:06.901957 1162 ts_tablet_manager.cc:1397] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.222s user 0.001s sys 0.009s
W20250115 14:52:06.902079 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 74d00bcee22d40769102eacaf728480d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74d00bcee22d40769102eacaf728480d already in progress: creating tablet
W20250115 14:52:06.902288 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9494c0c57be8499a883888d067c7db77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9494c0c57be8499a883888d067c7db77 already in progress: creating tablet
W20250115 14:52:06.902529 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 398ac8af619c427287b3964443173864 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 398ac8af619c427287b3964443173864 already in progress: creating tablet
I20250115 14:52:06.902680 1162 raft_consensus.cc:357] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.902866 1162 raft_consensus.cc:383] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.902922 1162 raft_consensus.cc:738] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.903097 1162 consensus_queue.cc:260] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.903544 1162 ts_tablet_manager.cc:1428] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:06.903836 1162 tablet_bootstrap.cc:492] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.754429 1751 rpcz_store.cc:276] 0115 14:52:05.581284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.586526 (+ 5242us) service_pool.cc:225] Handling call
0115 14:52:06.401039 (+814513us) ts_tablet_manager.cc:1145] Deleting tablet 8e6cc08065494d9e85eeff5044b44ba6
0115 14:52:06.401300 (+ 261us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.629079 (+227779us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.751475 (+122396us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.753776 (+ 2301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":727908}
I20250115 14:52:06.905457 1162 tablet_bootstrap.cc:654] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.906730 1168 tablet_bootstrap.cc:492] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:06.906898 1168 ts_tablet_manager.cc:1397] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.000s
I20250115 14:52:06.908574 882 tablet_service.cc:1467] Processing CreateTablet for tablet 792c68ba824640c1b653ad0bfbe2de02 (DEFAULT_TABLE table=default.Table_21e00049c7364adb9a87e33cd7a04957 [id=94f032d2cb6c42f9aea0538a13bed768]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.908993 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 792c68ba824640c1b653ad0bfbe2de02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.909138 1168 raft_consensus.cc:357] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.911402 1164 tablet_bootstrap.cc:492] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:06.911589 1164 ts_tablet_manager.cc:1397] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.155s user 0.007s sys 0.000s
I20250115 14:52:06.912397 1164 raft_consensus.cc:357] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.912684 1164 raft_consensus.cc:383] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.912786 1164 raft_consensus.cc:738] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.913040 1164 consensus_queue.cc:260] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.905503 1751 tablet_replica.cc:331] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:06.914709 1168 raft_consensus.cc:383] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.914750 2200 raft_consensus.cc:491] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.914960 2200 raft_consensus.cc:513] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.915902 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e924821d12045f6bc658461675aead4" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:06.916219 847 leader_election.cc:343] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 0e924821d12045f6bc658461675aead4
W20250115 14:52:06.916355 848 leader_election.cc:343] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.916457 848 leader_election.cc:304] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.916551 2225 raft_consensus.cc:2743] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.918078 1751 raft_consensus.cc:2238] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:06.918375 1751 raft_consensus.cc:2267] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:06.918826 1751 ts_tablet_manager.cc:1905] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:06.919180 2083 raft_consensus.cc:2743] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:06.919351 2083 consensus_queue.cc:1035] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:06.919904 2200 leader_election.cc:290] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:06.922322 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b126874bae6e4c84bd97f5ff65e0882b (DEFAULT_TABLE table=default.Table_340265c995a649808c582e4c4e6d0b22 [id=bf2a1ee254eb411dbbaccc0a78dfbad1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:06.928049 848 consensus_peers.cc:487] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:06.930482 876 tablet_service.cc:1467] Processing CreateTablet for tablet 064f890830304758a6840a1502e59080 (DEFAULT_TABLE table=default.Table_adcff7acc18b46e79d647e28af423fbd [id=d6f2abb6e8894bf58f3aa4876cc38ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.931754 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_892cf0050cb245a7af3e84a893239fba" } new_table_name: "default.Table_51bdcc69abfc456cb288efd2b3db4a3b" modify_external_catalogs: true
I20250115 14:52:06.932548 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_65de108282934cad838f2c1f716989a5" } new_table_name: "default.Table_c3e4c1d6466242fb8b2b5d97d504836b" modify_external_catalogs: true
I20250115 14:52:06.934357 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3950c52dccfb443bbfd1bed7a0007bd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.934859 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3950c52dccfb443bbfd1bed7a0007bd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:06.936947 2241 raft_consensus.cc:491] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:06.937096 2241 raft_consensus.cc:513] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.937687 2241 leader_election.cc:290] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.938311 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d590c63d6e4d058fac50dc50010317" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:06.938488 908 raft_consensus.cc:2463] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:06.939327 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d590c63d6e4d058fac50dc50010317" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:06.941083 983 leader_election.cc:304] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:52:06.941548 984 leader_election.cc:343] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.941874 2241 raft_consensus.cc:2798] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:06.941943 2241 raft_consensus.cc:491] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:06.941995 2241 raft_consensus.cc:3054] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.942245 1166 ts_tablet_manager.cc:1918] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:06.942337 1166 log.cc:1198] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/be63bae96c834a8bbbb1174c027935a2
I20250115 14:52:06.942582 1166 ts_tablet_manager.cc:1939] T be63bae96c834a8bbbb1174c027935a2 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:06.910375 891 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa67705596742c6b942f49bca0aa817 (DEFAULT_TABLE table=default.Table_3b9fd4f4e09a40e99f015f099eef8306 [id=254935fbfd31462c95a7762ab90fff86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.914857 1168 raft_consensus.cc:738] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.943383 1168 consensus_queue.cc:260] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.943912 1168 ts_tablet_manager.cc:1428] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.036s user 0.001s sys 0.000s
I20250115 14:52:06.944175 1168 tablet_bootstrap.cc:492] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:06.944825 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet be63bae96c834a8bbbb1174c027935a2 (table default.Table_705745b616194094bb03915f2f651122 [id=b2c6ee1ec113494cac7d1bf11a928714]) successfully deleted
I20250115 14:52:06.945523 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_df72eb78efd74af192eb92233075998c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.945631 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 105b6148ae944e5c906e07d987177cc5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 105b6148ae944e5c906e07d987177cc5 already in progress: creating tablet
W20250115 14:52:06.945896 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ca5cc55b50ee4953800f10efe0ffb546 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca5cc55b50ee4953800f10efe0ffb546 already in progress: creating tablet
W20250115 14:52:06.946038 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df72eb78efd74af192eb92233075998c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:06.946092 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 48b15de9bf8943cdb9e5abd943484993 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48b15de9bf8943cdb9e5abd943484993 already in progress: creating tablet
I20250115 14:52:06.946470 1168 tablet_bootstrap.cc:654] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.946473 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 010e6535b6bc44f391e6f2b6f1686532 (table default.Table_c5f747dc2f6b4cffb7b791ea6456c5b3 [id=13d88a401bed452688d5bb6f4dcfd70b]) successfully deleted
I20250115 14:52:06.946659 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet caa1827366a34f039fd12deb855f07b6 (table default.Table_a43e4255e5c3414aa47bff8bbe111d13 [id=aa88c2d281fe4856b6d6af91c9359f23]) successfully deleted
W20250115 14:52:06.946854 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 83415f1c07ad495cabd287bdcd54dad4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 83415f1c07ad495cabd287bdcd54dad4 already in progress: creating tablet
W20250115 14:52:06.947042 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a48a0af9422f43cf86531665c0fa1aee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a48a0af9422f43cf86531665c0fa1aee already in progress: creating tablet
W20250115 14:52:06.947227 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet cc9ce226d3ba44d6908257bbd0df3a12 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9ce226d3ba44d6908257bbd0df3a12 already in progress: creating tablet
I20250115 14:52:06.947814 1167 ts_tablet_manager.cc:1918] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250115 14:52:06.947839 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 already in progress: creating tablet
I20250115 14:52:06.947908 1167 log.cc:1198] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/009ddfe9dfe14ba2b067f907a9d13538
W20250115 14:52:06.948104 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet fa57c1fa96594a40b2551ad503127e4f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fa57c1fa96594a40b2551ad503127e4f already in progress: creating tablet
I20250115 14:52:06.948195 1167 ts_tablet_manager.cc:1939] T 009ddfe9dfe14ba2b067f907a9d13538 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:06.948469 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 064f890830304758a6840a1502e59080. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.948946 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa67705596742c6b942f49bca0aa817. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.950497 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_29a289769ec64b96a7352fa0412f999b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:06.951012 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29a289769ec64b96a7352fa0412f999b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:06.951181 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f7a5cbd9a78485a816d7fecdc23106a consensus state has no leader
I20250115 14:52:06.956349 2241 raft_consensus.cc:513] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:06.957489 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d590c63d6e4d058fac50dc50010317" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:06.958016 984 leader_election.cc:343] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:06.958724 749 tablet_service.cc:1467] Processing CreateTablet for tablet 7654d2831f0141858b52379b767b3a70 (DEFAULT_TABLE table=default.Table_2814144b33794c15b9386d76197e2e95 [id=d35c6b0e89914360b950527917ceef3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.959126 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7654d2831f0141858b52379b767b3a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.959093 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d590c63d6e4d058fac50dc50010317" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:06.959254 908 raft_consensus.cc:3054] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:06.960040 1162 tablet_bootstrap.cc:492] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:06.960152 1162 ts_tablet_manager.cc:1397] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.000s
I20250115 14:52:06.960934 2241 leader_election.cc:290] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:06.961234 2083 consensus_queue.cc:1035] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:06.961331 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 718) took 1666 ms. Trace:
I20250115 14:52:06.961393 1167 rpcz_store.cc:276] 0115 14:52:05.295151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.295201 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:06.816991 (+1521790us) ts_tablet_manager.cc:1145] Deleting tablet 009ddfe9dfe14ba2b067f907a9d13538
0115 14:52:06.817000 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.917548 (+100548us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.947702 (+ 30154us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:06.961316 (+ 13614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1492844}
I20250115 14:52:06.961730 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 719) took 1665 ms. Trace:
I20250115 14:52:06.962932 1162 raft_consensus.cc:357] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.964080 1162 raft_consensus.cc:383] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.964154 1162 raft_consensus.cc:738] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.964846 1162 consensus_queue.cc:260] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.965324 1162 ts_tablet_manager.cc:1428] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:52:06.965339 741 tablet_service.cc:1467] Processing CreateTablet for tablet e5dfc00ecec54f92a75fa77c84fe8a6d (DEFAULT_TABLE table=default.Table_5ce3a92ee6154dee8af40b3ac625550c [id=74a3f9e259514a4ab2ddfb67c1c0714c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.965574 1162 tablet_bootstrap.cc:492] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.965770 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5dfc00ecec54f92a75fa77c84fe8a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.969798 1162 tablet_bootstrap.cc:654] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.979808 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9727430a15574ec0b24a5a64da744cac (DEFAULT_TABLE table=default.Table_d4ae47e4d4274e66adc1753796b0ae1e [id=2ab741541b8d4f0abea120c6668dfae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.981086 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9727430a15574ec0b24a5a64da744cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.982926 1164 ts_tablet_manager.cc:1428] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.071s user 0.001s sys 0.000s
I20250115 14:52:06.983830 823 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:06.986106 748 tablet_service.cc:1467] Processing CreateTablet for tablet 4d79fb3826ff483ea6224fe3c277521c (DEFAULT_TABLE table=default.Table_519ee6346f5a476f875b3d4b5e249b0b [id=14152fc9c1264307a0e7a5e80735da62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.986434 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d79fb3826ff483ea6224fe3c277521c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.987891 1162 tablet_bootstrap.cc:492] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:06.988209 1162 ts_tablet_manager.cc:1397] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20250115 14:52:06.989401 1162 raft_consensus.cc:357] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.989682 1162 raft_consensus.cc:383] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.989790 1162 raft_consensus.cc:738] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.990115 1162 consensus_queue.cc:260] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.990971 1162 ts_tablet_manager.cc:1428] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250115 14:52:06.991235 740 tablet_service.cc:1467] Processing CreateTablet for tablet f38727d283924a14a5915a5010128d6e (DEFAULT_TABLE table=default.Table_6472f60e6915482395c4388061880777 [id=46847ce9d3584787b98ebeb9cdf62e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:06.991387 1167 rpcz_store.cc:276] 0115 14:52:05.296001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.296048 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:06.961624 (+1665576us) ts_tablet_manager.cc:1145] Deleting tablet db5518ac647c472bb47ed741022c820a
0115 14:52:06.961633 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.961715 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1637400}
I20250115 14:52:06.991539 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38727d283924a14a5915a5010128d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:06.991847 1164 tablet_bootstrap.cc:492] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:06.993535 1164 tablet_bootstrap.cc:654] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:06.994638 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 723) took 1547 ms. Trace:
I20250115 14:52:06.963497 908 raft_consensus.cc:2463] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:06.969132 1168 tablet_bootstrap.cc:492] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:06.994719 1167 rpcz_store.cc:276] 0115 14:52:05.446972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.455968 (+ 8996us) service_pool.cc:225] Handling call
0115 14:52:06.991656 (+1535688us) ts_tablet_manager.cc:1145] Deleting tablet a600fa0f0ec049dca2803bf7d2c74a95
0115 14:52:06.994503 (+ 2847us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.994618 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1535469}
I20250115 14:52:06.994887 1168 ts_tablet_manager.cc:1397] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.002s
I20250115 14:52:06.992489 1162 tablet_bootstrap.cc:492] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:06.995636 1168 raft_consensus.cc:357] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.995801 1168 raft_consensus.cc:383] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:06.995844 1168 raft_consensus.cc:738] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:06.996019 1168 consensus_queue.cc:260] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:06.996702 983 leader_election.cc:304] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:06.997467 1168 ts_tablet_manager.cc:1428] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:06.997745 2241 raft_consensus.cc:2798] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:06.997906 2241 raft_consensus.cc:695] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:06.998176 2241 consensus_queue.cc:237] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.000551 1168 tablet_bootstrap.cc:492] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.002090 1168 tablet_bootstrap.cc:654] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.004380 1167 tablet_replica.cc:331] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:07.004559 1167 raft_consensus.cc:2238] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:07.004892 1167 raft_consensus.cc:2267] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.005410 1167 ts_tablet_manager.cc:1905] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.009959 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 104282a09c024bbaafeb5f28f91dadcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.012398 1166 tablet_replica.cc:331] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.012573 1166 raft_consensus.cc:2238] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.012706 1166 raft_consensus.cc:2267] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.013688 1162 tablet_bootstrap.cc:654] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.014115 1166 ts_tablet_manager.cc:1905] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.015962 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b126874bae6e4c84bd97f5ff65e0882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.022079 752 tablet_service.cc:1467] Processing CreateTablet for tablet c26bb1836ee24d8fae9605d3c1818eef (DEFAULT_TABLE table=default.Table_d4b4b4d0c8e644acb0eaa155e8c59606 [id=314c91cf30bd4bfda8cf7b273edfc6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.025664 740 tablet_service.cc:1467] Processing CreateTablet for tablet db57f7c3a4c14cc19d3a0624e5b1817a (DEFAULT_TABLE table=default.Table_0b05d8cc5e384fccb6f6949ae2c33c98 [id=3140693ea82a44e986d02483b387cc4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.028507 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_adcff7acc18b46e79d647e28af423fbd" } modify_external_catalogs: true
I20250115 14:52:07.028738 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_adcff7acc18b46e79d647e28af423fbd" } modify_external_catalogs: true
I20250115 14:52:07.030710 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f3c56dc604f2471083ae4303c707c7e3" } modify_external_catalogs: true
I20250115 14:52:07.030862 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f3c56dc604f2471083ae4303c707c7e3" } modify_external_catalogs: true
W20250115 14:52:07.031564 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 83283.6us
I20250115 14:52:07.033278 450 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_4b79de2dd7fe493f84a77ae3c477e2c8 [id=471b029f1dba4dd087deb339bba02013]). Expected version 1 got 0
I20250115 14:52:07.035379 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_4b79de2dd7fe493f84a77ae3c477e2c8 [id=471b029f1dba4dd087deb339bba02013]). Expected version 1 got 0
I20250115 14:52:07.036365 455 master_service.cc:946] ReplaceTablet: received request to replace tablet b5937d6ad0494734a402fdc594eb9415 from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:07.039099 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4536bb3d44b40d2825e1f4e59d8a148 consensus state has no leader
I20250115 14:52:07.042532 751 tablet_service.cc:1467] Processing CreateTablet for tablet 894532f9a4d542a09eb51c2fcdd2907a (DEFAULT_TABLE table=default.Table_a75e911886994dffbae81cf1e8e94849 [id=a68990d66de442f8a3709cc02def1b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.042706 1167 ts_tablet_manager.cc:1918] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:07.042829 1167 log.cc:1198] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/dff155a5c2b0479a90e74993de6f402d
I20250115 14:52:07.045949 1167 ts_tablet_manager.cc:1939] T dff155a5c2b0479a90e74993de6f402d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:07.046430 748 tablet_service.cc:1467] Processing CreateTablet for tablet b3c31e24081d4a42aa933acbb80c6495 (DEFAULT_TABLE table=default.Table_f257047f59814e1a8458779252a782b9 [id=51d11f13250a43e8bc3b6988128d76d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.046959 749 tablet_service.cc:1467] Processing CreateTablet for tablet 6437873733584722979c1e9ae8b0730b (DEFAULT_TABLE table=default.Table_4dff2e869f8141e09dd757d6e47118f4 [id=43852683a5a64e1b85403395fa3d210c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.047433 741 tablet_service.cc:1467] Processing CreateTablet for tablet f2993e1c95234a1db2ae98a8e08c43f8 (DEFAULT_TABLE table=default.Table_d60ab02c55f94f3590d4b43fb3c8fa0a [id=c41d4b193ab34f1084a8b0b24578f05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.050521 1168 tablet_bootstrap.cc:492] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.050726 1168 ts_tablet_manager.cc:1397] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.003s
I20250115 14:52:07.051473 1168 raft_consensus.cc:357] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.051731 1168 raft_consensus.cc:383] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.051848 1168 raft_consensus.cc:738] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.052143 1168 consensus_queue.cc:260] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.052678 1168 ts_tablet_manager.cc:1428] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250115 14:52:07.059583 1168 tablet_bootstrap.cc:492] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.061269 1168 tablet_bootstrap.cc:654] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.063292 1162 tablet_bootstrap.cc:492] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.063429 1162 ts_tablet_manager.cc:1397] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.072s user 0.001s sys 0.005s
I20250115 14:52:07.064038 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 732) took 1616 ms. Trace:
I20250115 14:52:07.064124 1162 raft_consensus.cc:357] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.064325 1162 raft_consensus.cc:383] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.064390 1162 raft_consensus.cc:738] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.064610 1162 consensus_queue.cc:260] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.065150 1162 ts_tablet_manager.cc:1428] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:07.064616 1167 rpcz_store.cc:276] 0115 14:52:05.447747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.456354 (+ 8607us) service_pool.cc:225] Handling call
0115 14:52:06.994937 (+1538583us) ts_tablet_manager.cc:1145] Deleting tablet dff155a5c2b0479a90e74993de6f402d
0115 14:52:07.000320 (+ 5383us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.008162 (+ 7842us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.042613 (+ 34451us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.064022 (+ 21409us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1538430}
I20250115 14:52:07.065397 1162 tablet_bootstrap.cc:492] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:07.065721 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 727) took 1618 ms. Trace:
I20250115 14:52:07.066886 1162 tablet_bootstrap.cc:654] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.065966 1164 tablet_bootstrap.cc:492] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.067181 1164 ts_tablet_manager.cc:1397] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.075s user 0.006s sys 0.003s
I20250115 14:52:07.068101 1164 raft_consensus.cc:357] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.069257 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894532f9a4d542a09eb51c2fcdd2907a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.072516 1164 raft_consensus.cc:383] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.072580 1164 raft_consensus.cc:738] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.072981 1164 consensus_queue.cc:260] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.076735 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c26bb1836ee24d8fae9605d3c1818eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.077729 1162 tablet_bootstrap.cc:492] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.077839 1162 ts_tablet_manager.cc:1397] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20250115 14:52:07.078470 1162 raft_consensus.cc:357] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.078629 1162 raft_consensus.cc:383] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.078672 1162 raft_consensus.cc:738] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.078878 1162 consensus_queue.cc:260] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.079324 1162 ts_tablet_manager.cc:1428] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250115 14:52:07.079345 1164 ts_tablet_manager.cc:1428] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250115 14:52:07.079543 1162 tablet_bootstrap.cc:492] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:07.079582 1164 tablet_bootstrap.cc:492] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.080968 1162 tablet_bootstrap.cc:654] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.081068 1164 tablet_bootstrap.cc:654] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.082258 2241 raft_consensus.cc:491] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.082369 2241 raft_consensus.cc:513] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.082968 2241 leader_election.cc:290] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.083561 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600fa0f0ec049dca2803bf7d2c74a95" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:07.083627 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600fa0f0ec049dca2803bf7d2c74a95" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:07.084070 984 leader_election.cc:343] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:07.084213 983 leader_election.cc:343] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.084319 983 leader_election.cc:304] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.084812 2241 raft_consensus.cc:2743] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.087327 1168 tablet_bootstrap.cc:492] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.087493 1168 ts_tablet_manager.cc:1397] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.005s
I20250115 14:52:07.088186 1168 raft_consensus.cc:357] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.088429 1168 raft_consensus.cc:383] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.088515 1168 raft_consensus.cc:738] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.088757 1168 consensus_queue.cc:260] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.089304 1168 ts_tablet_manager.cc:1428] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:07.089579 1168 tablet_bootstrap.cc:492] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.075264 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 5bc975b044dd439dbd1f8f9a82cce751 (table default.Table_124bdb85e4194ff5a214a6b79870a1cf [id=f8ea853486b54a39843ef2b5628fbe37]) successfully deleted
W20250115 14:52:07.090425 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a2bcfc5c5714a3eac85b6a8beafbf59 already in progress: creating tablet
W20250115 14:52:07.090731 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet fb81e8fcc38a41e69cfb0e34dbaaac84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fb81e8fcc38a41e69cfb0e34dbaaac84 already in progress: creating tablet
W20250115 14:52:07.091027 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d1259256b7744e739581125611abbe67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1259256b7744e739581125611abbe67 already in progress: creating tablet
I20250115 14:52:07.091081 1168 tablet_bootstrap.cc:654] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:07.091300 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 65a6f83730f74d13823bd15d93538df7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65a6f83730f74d13823bd15d93538df7 already in progress: creating tablet
W20250115 14:52:07.091524 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 57aecd22f97c41618f90d14c2f53a7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57aecd22f97c41618f90d14c2f53a7c9 already in progress: creating tablet
W20250115 14:52:07.091771 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet be8e12ccb8d14c1c9258dc3f81d1527d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be8e12ccb8d14c1c9258dc3f81d1527d already in progress: creating tablet
W20250115 14:52:07.092203 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 567365040b6d4d6eba04112c8d4c3dfb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 567365040b6d4d6eba04112c8d4c3dfb already in progress: creating tablet
I20250115 14:52:07.092341 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db57f7c3a4c14cc19d3a0624e5b1817a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.093129 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 009ddfe9dfe14ba2b067f907a9d13538 (table default.Table_96eab6cd20b842138b5acc7c8d0ac5d6 [id=816d14738e72406a88b9eed42d72be73]) successfully deleted
W20250115 14:52:07.093485 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet db5518ac647c472bb47ed741022c820a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db5518ac647c472bb47ed741022c820a already in progress: creating tablet
W20250115 14:52:07.093667 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a600fa0f0ec049dca2803bf7d2c74a95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a600fa0f0ec049dca2803bf7d2c74a95 already in progress: creating tablet
I20250115 14:52:07.093799 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet dff155a5c2b0479a90e74993de6f402d (table default.Table_49688b4238ca4abaac09f3cbdb738761 [id=f6da20956eb349709bbe9e43a41fe4ed]) successfully deleted
I20250115 14:52:07.094389 891 tablet_service.cc:1514] Processing DeleteTablet for tablet ba33eb3f5a40450db5cf115fbe8a17e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.094648 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07686c40324233a49e4c49aeed73f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.094830 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b803d8e103a5423f85c75c9006809b17 with delete_type TABLET_DATA_DELETED (replaced by tablet 6437873733584722979c1e9ae8b0730b) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.095211 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 580742eb5dac4dba9995809890e0e315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.095194 1015 tablet_service.cc:1467] Processing CreateTablet for tablet a6270d3c4ee94679a92f8bbfd1aa5d95 (DEFAULT_TABLE table=default.Table_e37f4580bc3c4d5cacb53a430d3d5605 [id=6087ca0bad3147b9ace51f422882725c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.095285 885 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 with delete_type TABLET_DATA_DELETED (replaced by tablet cd8586f571634e678e783e32d34f887f) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.095343 1029 tablet_service.cc:1467] Processing CreateTablet for tablet b4248afe06a5400ca0ce38549f82b400 (DEFAULT_TABLE table=default.Table_572841278ecd4e1d9c55e42ea357b3c1 [id=6547c744e1354980ac24674b78afd741]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.095528 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6270d3c4ee94679a92f8bbfd1aa5d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.095521 879 tablet_service.cc:1467] Processing CreateTablet for tablet b4248afe06a5400ca0ce38549f82b400 (DEFAULT_TABLE table=default.Table_572841278ecd4e1d9c55e42ea357b3c1 [id=6547c744e1354980ac24674b78afd741]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.095587 880 tablet_service.cc:1467] Processing CreateTablet for tablet ad417f06081d41248dff6f1eed175f29 (DEFAULT_TABLE table=default.Table_f047206807ce42eb89594d08a8209682 [id=083c86d9313c40f590d6840429fe1624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.095809 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4248afe06a5400ca0ce38549f82b400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.095927 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet ba33eb3f5a40450db5cf115fbe8a17e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.096132 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad417f06081d41248dff6f1eed175f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.096668 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4248afe06a5400ca0ce38549f82b400. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.100534 1018 tablet_service.cc:1467] Processing CreateTablet for tablet b579be5c683e4fd8a5d5484fdb89250c (DEFAULT_TABLE table=default.Table_2179cfd883084fe7864e4cf7bee57581 [id=caea68b0c11f4acebe8b53aea12711df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.100836 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b579be5c683e4fd8a5d5484fdb89250c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.102110 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c31e24081d4a42aa933acbb80c6495. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.103392 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 580742eb5dac4dba9995809890e0e315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.104498 1014 tablet_service.cc:1467] Processing CreateTablet for tablet ad417f06081d41248dff6f1eed175f29 (DEFAULT_TABLE table=default.Table_f047206807ce42eb89594d08a8209682 [id=083c86d9313c40f590d6840429fe1624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.104758 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad417f06081d41248dff6f1eed175f29. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.105955 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:40227, user_credentials={real_user=slave}} blocked reactor thread for 77229us
I20250115 14:52:07.108422 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6437873733584722979c1e9ae8b0730b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.108834 893 tablet_service.cc:1467] Processing CreateTablet for tablet b579be5c683e4fd8a5d5484fdb89250c (DEFAULT_TABLE table=default.Table_2179cfd883084fe7864e4cf7bee57581 [id=caea68b0c11f4acebe8b53aea12711df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.109118 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b579be5c683e4fd8a5d5484fdb89250c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.109261 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 78a2d39d0296468db48b3eec374244ca (table default.Table_0f45c2d986f34bb7ae3fcd712eb1548f [id=4427bf7f81e345b1b5e5a07c81073d08], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 78a2d39d0296468db48b3eec374244ca consensus state has no leader
I20250115 14:52:07.110965 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet b803d8e103a5423f85c75c9006809b17 with delete_type TABLET_DATA_DELETED (replaced by tablet 6437873733584722979c1e9ae8b0730b) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.111124 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a059e2f0654a4aafe9e8082b8fde7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.111021 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1692c29b37a04b5697933d4456c5319e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.111564 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1692c29b37a04b5697933d4456c5319e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.112803 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_095514432af8489eab6e04a8b9bf1f53" } new_table_name: "default.Table_7a6e5fd036c94dcd843bea04f7377aee" modify_external_catalogs: true
I20250115 14:52:07.113932 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e8f8acbb30db4f019ebeadda783b434a" } modify_external_catalogs: true
I20250115 14:52:07.114080 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e8f8acbb30db4f019ebeadda783b434a" } modify_external_catalogs: true
I20250115 14:52:07.114456 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet ded663ac1197454bbde3c4c2d526002d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.115726 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0f45c2d986f34bb7ae3fcd712eb1548f" } modify_external_catalogs: true
I20250115 14:52:07.115878 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0f45c2d986f34bb7ae3fcd712eb1548f" } modify_external_catalogs: true
I20250115 14:52:07.116971 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d43d582508924d259b43a911c5d2c876" } new_table_name: "default.Table_f1bd3a4a8eb5403ca64cf59d41757e64" modify_external_catalogs: true
I20250115 14:52:07.118347 891 tablet_service.cc:1467] Processing CreateTablet for tablet a6270d3c4ee94679a92f8bbfd1aa5d95 (DEFAULT_TABLE table=default.Table_e37f4580bc3c4d5cacb53a430d3d5605 [id=6087ca0bad3147b9ace51f422882725c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.118656 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6270d3c4ee94679a92f8bbfd1aa5d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.123286 1751 ts_tablet_manager.cc:1918] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:07.123436 1751 log.cc:1198] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/171643c40c9b43ffb823f7c33789284d
I20250115 14:52:07.123461 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_40e515dbe043466f8d1ca690f14c6f42" } modify_external_catalogs: true
I20250115 14:52:07.123641 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_40e515dbe043466f8d1ca690f14c6f42" } modify_external_catalogs: true
I20250115 14:52:07.124001 1751 ts_tablet_manager.cc:1939] T 171643c40c9b43ffb823f7c33789284d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:07.124802 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 775) took 1317 ms. Trace:
I20250115 14:52:07.124943 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6e2d55a4e32448fbb98f704dbf1657b9
I20250115 14:52:07.124878 1751 rpcz_store.cc:276] 0115 14:52:05.807699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.807898 (+ 199us) service_pool.cc:225] Handling call
0115 14:52:06.905455 (+1097557us) ts_tablet_manager.cc:1145] Deleting tablet 171643c40c9b43ffb823f7c33789284d
0115 14:52:06.905466 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:06.918264 (+ 12798us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x616000196354. stack: 00007f9adcf568a6 00007f9adcf56660 00007f9adbf3b48d 000000000096d544 000000000096d50d 000000000096ceda 00007f9ae87f7ed0 00007f9ae9aec463 00007f9aec590f67 00007f9aec5c3634 00007f9aec5a4740 00007f9aec5a44cd 0000000000a71ecc 00007f9adcf9d434 00007f9adcfa0a5b 00007f9adcfa08d8
0115 14:52:07.017082 (+ 98818us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.123212 (+106130us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.124781 (+ 1569us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7757696,"tablet-delete.queue_time_us":1097311}
I20250115 14:52:07.125463 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 778) took 1265 ms. Trace:
I20250115 14:52:07.125532 1751 rpcz_store.cc:276] 0115 14:52:05.860321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.860366 (+ 45us) service_pool.cc:225] Handling call
0115 14:52:07.125327 (+1264961us) ts_tablet_manager.cc:1145] Deleting tablet fbf5da59db79418498db09ec8ea3ac0a
0115 14:52:07.125338 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.125447 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264795}
I20250115 14:52:07.125546 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3caaaaf71eae487480e4b55631f7e099"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.126065 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3caaaaf71eae487480e4b55631f7e099 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.125980 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 776) took 1269 ms. Trace:
I20250115 14:52:07.126143 1751 rpcz_store.cc:276] 0115 14:52:05.856540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.856619 (+ 79us) service_pool.cc:225] Handling call
0115 14:52:07.125847 (+1269228us) ts_tablet_manager.cc:1145] Deleting tablet 35301651222249d0b3439e0638f599f2
0115 14:52:07.125856 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.125966 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264210}
I20250115 14:52:07.126554 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 777) took 1266 ms. Trace:
I20250115 14:52:07.126617 1751 rpcz_store.cc:276] 0115 14:52:05.860211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.862008 (+ 1797us) service_pool.cc:225] Handling call
0115 14:52:07.126440 (+1264432us) ts_tablet_manager.cc:1145] Deleting tablet e59b36e986e146b9980c1f71599f32aa
0115 14:52:07.126449 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.126540 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264291}
I20250115 14:52:07.126945 1751 tablet_replica.cc:331] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:07.127125 1751 raft_consensus.cc:2238] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.127130 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b5937d6ad0494734a402fdc594eb9415
I20250115 14:52:07.127250 1751 raft_consensus.cc:2267] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.127569 455 catalog_manager.cc:6596] ReplaceTablet: tablet b5937d6ad0494734a402fdc594eb9415 deleted and replaced by tablet f2ba4086a6bf4bfa83e740b31dfb53f8
I20250115 14:52:07.127705 1751 ts_tablet_manager.cc:1905] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.127918 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 064f890830304758a6840a1502e59080
I20250115 14:52:07.136611 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2993e1c95234a1db2ae98a8e08c43f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.149581 1168 tablet_bootstrap.cc:492] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.149773 1168 ts_tablet_manager.cc:1397] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.002s
I20250115 14:52:07.150713 1168 raft_consensus.cc:357] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.150959 1168 raft_consensus.cc:383] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.151042 1168 raft_consensus.cc:738] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.151257 1168 consensus_queue.cc:260] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.069794 1166 ts_tablet_manager.cc:1918] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.152429 1166 log.cc:1198] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/104282a09c024bbaafeb5f28f91dadcb
I20250115 14:52:07.152774 1166 ts_tablet_manager.cc:1939] T 104282a09c024bbaafeb5f28f91dadcb P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.154654 1168 ts_tablet_manager.cc:1428] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:07.154950 1168 tablet_bootstrap.cc:492] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.156420 1168 tablet_bootstrap.cc:654] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.157130 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7b686a6b4043b78a9a59292bcc34ba (DEFAULT_TABLE table=default.Table_3e03f479683449e197a5150099b49504 [id=6801ef0ff2474c348248d37b86192981]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.157469 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7b686a6b4043b78a9a59292bcc34ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.153733 886 tablet_service.cc:1514] Processing DeleteTablet for tablet ded663ac1197454bbde3c4c2d526002d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.156818 879 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7b686a6b4043b78a9a59292bcc34ba (DEFAULT_TABLE table=default.Table_3e03f479683449e197a5150099b49504 [id=6801ef0ff2474c348248d37b86192981]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.160163 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7b686a6b4043b78a9a59292bcc34ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.165160 891 tablet_service.cc:1467] Processing CreateTablet for tablet 11aeb6c73b4d42058736ffce551e311c (DEFAULT_TABLE table=default.Table_69ed5f68465549b38e7e27bf0d75eb2e [id=f08c5a8508224d27a45438fe8bd9390e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.165459 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11aeb6c73b4d42058736ffce551e311c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.165478 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 11aeb6c73b4d42058736ffce551e311c (DEFAULT_TABLE table=default.Table_69ed5f68465549b38e7e27bf0d75eb2e [id=f08c5a8508224d27a45438fe8bd9390e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.165817 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11aeb6c73b4d42058736ffce551e311c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.171568 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2d55a4e32448fbb98f704dbf1657b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.172884 749 tablet_service.cc:1514] Processing DeleteTablet for tablet b5937d6ad0494734a402fdc594eb9415 with delete_type TABLET_DATA_DELETED (replaced by tablet f2ba4086a6bf4bfa83e740b31dfb53f8) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.173231 1164 tablet_bootstrap.cc:492] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.173394 1164 ts_tablet_manager.cc:1397] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.094s user 0.000s sys 0.004s
I20250115 14:52:07.174046 1164 raft_consensus.cc:357] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.174274 1164 raft_consensus.cc:383] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.174360 1164 raft_consensus.cc:738] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.174602 1164 consensus_queue.cc:260] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.175109 1164 ts_tablet_manager.cc:1428] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:07.175804 1164 tablet_bootstrap.cc:492] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.177345 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 064f890830304758a6840a1502e59080 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.178114 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 104282a09c024bbaafeb5f28f91dadcb (table default.Table_06a952903c7b4dcc8088b8c9ff5a2ac5 [id=09205f90c7fc476185a1d31a60ad5fa0]) successfully deleted
W20250115 14:52:07.178582 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6e2d55a4e32448fbb98f704dbf1657b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2d55a4e32448fbb98f704dbf1657b9 already in progress: creating tablet
I20250115 14:52:07.178750 1164 tablet_bootstrap.cc:654] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:07.178887 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b5937d6ad0494734a402fdc594eb9415 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5937d6ad0494734a402fdc594eb9415 already in progress: creating tablet
W20250115 14:52:07.179083 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 064f890830304758a6840a1502e59080 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 064f890830304758a6840a1502e59080 already in progress: creating tablet
I20250115 14:52:07.179476 454 master_service.cc:946] ReplaceTablet: received request to replace tablet f5c1f0e0b6ee4448b9ba2f78d0117197 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:07.180696 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_86158e4855654f77bfa28259f0d0e38b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.181211 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86158e4855654f77bfa28259f0d0e38b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.183207 960 heartbeater.cc:346] Connected to a master server at 127.0.104.61:45653
I20250115 14:52:07.184610 1162 tablet_bootstrap.cc:492] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.184751 1162 ts_tablet_manager.cc:1397] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.105s user 0.012s sys 0.012s
I20250115 14:52:07.185457 1162 raft_consensus.cc:357] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.185622 1162 raft_consensus.cc:383] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.185670 1162 raft_consensus.cc:738] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.185855 1162 consensus_queue.cc:260] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.186298 1162 ts_tablet_manager.cc:1428] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:07.186671 1162 tablet_bootstrap.cc:492] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:07.187901 1162 tablet_bootstrap.cc:654] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.195186 886 tablet_service.cc:1467] Processing CreateTablet for tablet 00e6cb19641f49e599275a5a5fc272fb (DEFAULT_TABLE table=default.Table_b02e318b61ed4745bf6bf686cd720090 [id=aaa486d6ed38402a98803f39ad3e5c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.195634 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e6cb19641f49e599275a5a5fc272fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.197832 2273 raft_consensus.cc:491] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.197944 2273 raft_consensus.cc:513] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.198474 2273 leader_election.cc:290] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.198937 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e95d09efd2e44cc81cdcbf87ee453fe" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:07.199574 983 leader_election.cc:343] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.200224 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e95d09efd2e44cc81cdcbf87ee453fe" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:07.200798 984 leader_election.cc:343] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.200894 984 leader_election.cc:304] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.201264 2273 raft_consensus.cc:2743] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:07.204528 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5882252383ba46aba0ec053fab1fd383 consensus state has no leader
I20250115 14:52:07.204910 1162 tablet_bootstrap.cc:492] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.205010 1162 ts_tablet_manager.cc:1397] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.001s
I20250115 14:52:07.204922 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_13267a7581594c869ec943c3a7f1058e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.205451 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13267a7581594c869ec943c3a7f1058e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.205642 1162 raft_consensus.cc:357] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.205811 1162 raft_consensus.cc:383] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.205857 1162 raft_consensus.cc:738] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.206053 1162 consensus_queue.cc:260] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.206504 1162 ts_tablet_manager.cc:1428] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250115 14:52:07.206605 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet efb8b001dd66456d9e49e9e8a6848ccf
I20250115 14:52:07.206810 1164 tablet_bootstrap.cc:492] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.206921 1164 ts_tablet_manager.cc:1397] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250115 14:52:07.207224 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 78a2d39d0296468db48b3eec374244ca
I20250115 14:52:07.207536 1164 raft_consensus.cc:357] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.207697 1164 raft_consensus.cc:383] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.207746 1164 raft_consensus.cc:738] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
W20250115 14:52:07.207859 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba21cf54bc074d95b456fd70a9286001 consensus state has no leader
I20250115 14:52:07.208590 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f5c1f0e0b6ee4448b9ba2f78d0117197
I20250115 14:52:07.208964 454 catalog_manager.cc:6596] ReplaceTablet: tablet f5c1f0e0b6ee4448b9ba2f78d0117197 deleted and replaced by tablet 8c545ef40b4743c89b9e9165982d4b2f
I20250115 14:52:07.208927 1167 rpcz_store.cc:276] 0115 14:52:05.447335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.456715 (+ 9380us) service_pool.cc:225] Handling call
0115 14:52:07.065591 (+1608876us) ts_tablet_manager.cc:1145] Deleting tablet e9febafb7e354e289cd3f864311f14cb
0115 14:52:07.065600 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.065682 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1608646}
I20250115 14:52:07.209396 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 724) took 1762 ms. Trace:
I20250115 14:52:07.209494 1167 rpcz_store.cc:276] 0115 14:52:05.447080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.463551 (+ 16471us) service_pool.cc:225] Handling call
0115 14:52:07.209292 (+1745741us) ts_tablet_manager.cc:1145] Deleting tablet 36514c5399924e98b3bc38004a33b919
0115 14:52:07.209301 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.209381 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745585}
I20250115 14:52:07.209895 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 45f124134eab4babac145404c62e7d2a
I20250115 14:52:07.209867 1164 consensus_queue.cc:260] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.210325 1164 ts_tablet_manager.cc:1428] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:07.210548 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 734) took 1741 ms. Trace:
I20250115 14:52:07.210973 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 00e6cb19641f49e599275a5a5fc272fb (DEFAULT_TABLE table=default.Table_b02e318b61ed4745bf6bf686cd720090 [id=aaa486d6ed38402a98803f39ad3e5c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.211709 879 tablet_service.cc:1467] Processing CreateTablet for tablet d360add9d39a48caa7b90f7d786f7bc1 (DEFAULT_TABLE table=default.Table_76b7b4a0c17041068bfb2be07cf21267 [id=df847f2fc5394b74832182b18e3e1701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.212134 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d360add9d39a48caa7b90f7d786f7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.212414 1014 tablet_service.cc:1467] Processing CreateTablet for tablet ff771f5acdcf4fcb9e099aede920f0a1 (DEFAULT_TABLE table=default.Table_b1ed6d26296a4f9799945452887c3c1c [id=0a128de980814ba281cf1bc3b6e26fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.212720 2200 raft_consensus.cc:491] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.212862 891 tablet_service.cc:1467] Processing CreateTablet for tablet ff771f5acdcf4fcb9e099aede920f0a1 (DEFAULT_TABLE table=default.Table_b1ed6d26296a4f9799945452887c3c1c [id=0a128de980814ba281cf1bc3b6e26fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.213238 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff771f5acdcf4fcb9e099aede920f0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.213385 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c1602aef514dd29d5d2cffb043dbb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.213718 1018 tablet_service.cc:1467] Processing CreateTablet for tablet d360add9d39a48caa7b90f7d786f7bc1 (DEFAULT_TABLE table=default.Table_76b7b4a0c17041068bfb2be07cf21267 [id=df847f2fc5394b74832182b18e3e1701]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.212839 2200 raft_consensus.cc:513] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.216763 2200 leader_election.cc:290] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.217562 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 282d988bc41443c381bb0ba69b618f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 567365040b6d4d6eba04112c8d4c3dfb) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:07.217622 847 leader_election.cc:343] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 01291d41a2b345b4bfd00e6be0fb0129
I20250115 14:52:07.218067 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 79736602ea2646b39708ca9c6834e847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.218269 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 79736602ea2646b39708ca9c6834e847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.219317 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 40eae66b35974524992b2ef943273291 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.219568 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 40eae66b35974524992b2ef943273291 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.225416 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01291d41a2b345b4bfd00e6be0fb0129" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:07.225531 1164 tablet_bootstrap.cc:492] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:07.225929 848 leader_election.cc:343] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.226035 848 leader_election.cc:304] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.226230 2200 raft_consensus.cc:2743] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.226392 1167 rpcz_store.cc:276] 0115 14:52:05.469144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.469187 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:07.210456 (+1741269us) ts_tablet_manager.cc:1145] Deleting tablet 6ce498ffd3f04c5d8eb642739a838903
0115 14:52:07.210464 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.210534 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741061}
I20250115 14:52:07.227443 1167 tablet_replica.cc:331] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:07.227658 1167 raft_consensus.cc:2238] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.227881 1167 raft_consensus.cc:2267] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.232772 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c0d7ba16034c431d8c39b3ca49d63d88" } new_table_name: "default.Table_d611f6594f554cfe99c5aa5f2f7ea0e2" modify_external_catalogs: true
I20250115 14:52:07.227270 1168 tablet_bootstrap.cc:492] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.234243 1168 ts_tablet_manager.cc:1397] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.000s
I20250115 14:52:07.234943 1168 raft_consensus.cc:357] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.235158 1168 raft_consensus.cc:383] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.235239 1168 raft_consensus.cc:738] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.235448 1168 consensus_queue.cc:260] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.236079 1168 ts_tablet_manager.cc:1428] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:07.236369 1168 tablet_bootstrap.cc:492] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.237602 1168 tablet_bootstrap.cc:654] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.238840 1167 ts_tablet_manager.cc:1905] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.239410 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e924821d12045f6bc658461675aead4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.239639 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ff71ded3b074ff0979e2090131f4eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.239825 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ff71ded3b074ff0979e2090131f4eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.240567 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7f957146690846bd9bf51f27431bd54d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:07.240936 2273 raft_consensus.cc:491] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:52:07.241076 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f957146690846bd9bf51f27431bd54d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.241052 2273 raft_consensus.cc:513] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.241577 2273 leader_election.cc:290] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.241905 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa57c1fa96594a40b2551ad503127e4f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:07.242022 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa57c1fa96594a40b2551ad503127e4f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:07.242367 984 leader_election.cc:343] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:07.242519 983 leader_election.cc:343] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.242604 983 leader_election.cc:304] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.243093 2273 raft_consensus.cc:2743] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.243294 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b7f7672a657044a38068152b29444a85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:07.243419 1164 tablet_bootstrap.cc:654] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:07.243714 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7f7672a657044a38068152b29444a85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.246611 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_236720cf03dd4cf7b5e1943e11dbe3d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.247560 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_236720cf03dd4cf7b5e1943e11dbe3d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.248701 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 72a97d102c174bc3a5e3a3c5762033b3 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:07.249584 749 tablet_service.cc:1514] Processing DeleteTablet for tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 with delete_type TABLET_DATA_DELETED (replaced by tablet cd8586f571634e678e783e32d34f887f) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.249856 1166 tablet_replica.cc:331] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.250020 1166 raft_consensus.cc:2238] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.250146 1166 raft_consensus.cc:2267] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.251181 1166 ts_tablet_manager.cc:1905] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.252633 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 338706b2d7514f94afd3579da458b4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.252862 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 338706b2d7514f94afd3579da458b4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.253226 1164 tablet_bootstrap.cc:492] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.253376 879 tablet_service.cc:1514] Processing DeleteTablet for tablet b579be5c683e4fd8a5d5484fdb89250c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.253703 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet b579be5c683e4fd8a5d5484fdb89250c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.254525 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9727430a15574ec0b24a5a64da744cac (DEFAULT_TABLE table=default.Table_d4ae47e4d4274e66adc1753796b0ae1e [id=2ab741541b8d4f0abea120c6668dfae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.254909 879 tablet_service.cc:1467] Processing CreateTablet for tablet 9727430a15574ec0b24a5a64da744cac (DEFAULT_TABLE table=default.Table_d4ae47e4d4274e66adc1753796b0ae1e [id=2ab741541b8d4f0abea120c6668dfae2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.255290 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9727430a15574ec0b24a5a64da744cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.256007 1010 tablet_service.cc:1467] Processing CreateTablet for tablet e5dfc00ecec54f92a75fa77c84fe8a6d (DEFAULT_TABLE table=default.Table_5ce3a92ee6154dee8af40b3ac625550c [id=74a3f9e259514a4ab2ddfb67c1c0714c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.257045 893 tablet_service.cc:1467] Processing CreateTablet for tablet 7654d2831f0141858b52379b767b3a70 (DEFAULT_TABLE table=default.Table_2814144b33794c15b9386d76197e2e95 [id=d35c6b0e89914360b950527917ceef3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.257349 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7654d2831f0141858b52379b767b3a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.257289 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 7654d2831f0141858b52379b767b3a70 (DEFAULT_TABLE table=default.Table_2814144b33794c15b9386d76197e2e95 [id=d35c6b0e89914360b950527917ceef3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.257860 880 tablet_service.cc:1467] Processing CreateTablet for tablet 4d79fb3826ff483ea6224fe3c277521c (DEFAULT_TABLE table=default.Table_519ee6346f5a476f875b3d4b5e249b0b [id=14152fc9c1264307a0e7a5e80735da62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.258147 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d79fb3826ff483ea6224fe3c277521c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.258147 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 4d79fb3826ff483ea6224fe3c277521c (DEFAULT_TABLE table=default.Table_519ee6346f5a476f875b3d4b5e249b0b [id=14152fc9c1264307a0e7a5e80735da62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.258692 882 tablet_service.cc:1467] Processing CreateTablet for tablet f38727d283924a14a5915a5010128d6e (DEFAULT_TABLE table=default.Table_6472f60e6915482395c4388061880777 [id=46847ce9d3584787b98ebeb9cdf62e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.258960 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38727d283924a14a5915a5010128d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.255879 891 tablet_service.cc:1467] Processing CreateTablet for tablet e5dfc00ecec54f92a75fa77c84fe8a6d (DEFAULT_TABLE table=default.Table_5ce3a92ee6154dee8af40b3ac625550c [id=74a3f9e259514a4ab2ddfb67c1c0714c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.262750 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5dfc00ecec54f92a75fa77c84fe8a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.264571 1028 tablet_service.cc:1467] Processing CreateTablet for tablet f38727d283924a14a5915a5010128d6e (DEFAULT_TABLE table=default.Table_6472f60e6915482395c4388061880777 [id=46847ce9d3584787b98ebeb9cdf62e1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.264678 1162 tablet_bootstrap.cc:492] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:07.265014 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38727d283924a14a5915a5010128d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.266188 1162 tablet_bootstrap.cc:654] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.268155 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff771f5acdcf4fcb9e099aede920f0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.271945 1164 ts_tablet_manager.cc:1397] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.002s
I20250115 14:52:07.272609 1164 raft_consensus.cc:357] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.272785 1164 raft_consensus.cc:383] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.272832 1164 raft_consensus.cc:738] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.273046 1164 consensus_queue.cc:260] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.258468 1166 ts_tablet_manager.cc:1918] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.258936 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 104282a09c024bbaafeb5f28f91dadcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.261169 741 tablet_service.cc:1514] Processing DeleteTablet for tablet efb8b001dd66456d9e49e9e8a6848ccf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.273397 1166 log.cc:1198] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/fc3d2ca7cd7c48a79ed1396fbbdc6866
I20250115 14:52:07.261720 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a2d39d0296468db48b3eec374244ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.271147 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c1f0e0b6ee4448b9ba2f78d0117197 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c545ef40b4743c89b9e9165982d4b2f) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.273667 1166 ts_tablet_manager.cc:1939] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.275625 1167 ts_tablet_manager.cc:1918] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:07.275725 1167 log.cc:1198] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/171643c40c9b43ffb823f7c33789284d
I20250115 14:52:07.276044 1167 ts_tablet_manager.cc:1939] T 171643c40c9b43ffb823f7c33789284d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:07.276602 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f124134eab4babac145404c62e7d2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.280966 1751 ts_tablet_manager.cc:1918] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.281069 1751 log.cc:1198] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4341977f363d40eeac71b7d33d1b43ac
I20250115 14:52:07.281320 1751 ts_tablet_manager.cc:1939] T 4341977f363d40eeac71b7d33d1b43ac P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:07.281944 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 779) took 1403 ms. Trace:
I20250115 14:52:07.282071 1164 ts_tablet_manager.cc:1428] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250115 14:52:07.282034 1751 rpcz_store.cc:276] 0115 14:52:05.878590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.878645 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:07.126904 (+1248259us) ts_tablet_manager.cc:1145] Deleting tablet 4341977f363d40eeac71b7d33d1b43ac
0115 14:52:07.126913 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.257023 (+130110us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.280900 (+ 23877us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.281927 (+ 1027us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1248075}
I20250115 14:52:07.282544 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9727430a15574ec0b24a5a64da744cac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.283383 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5dfc00ecec54f92a75fa77c84fe8a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.283401 1168 tablet_bootstrap.cc:492] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.283514 1168 ts_tablet_manager.cc:1397] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20250115 14:52:07.284112 1168 raft_consensus.cc:357] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.284273 1168 raft_consensus.cc:383] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.284351 1168 raft_consensus.cc:738] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.284511 1168 consensus_queue.cc:260] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.284924 1168 ts_tablet_manager.cc:1428] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250115 14:52:07.285178 1168 tablet_bootstrap.cc:492] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.286123 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7654d2831f0141858b52379b767b3a70. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.286546 1011 consensus_meta.cc:287] T 9ad3cda1056349a789a1751c5d386d0c P c238c037a1c74725a001e2b4a1ff4304: Time spent flushing consensus metadata: real 0.556s user 0.016s sys 0.006s
I20250115 14:52:07.288504 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d79fb3826ff483ea6224fe3c277521c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.289724 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 (table default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]) successfully deleted
W20250115 14:52:07.289942 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 104282a09c024bbaafeb5f28f91dadcb because the tablet was not found. No further retry: Not found: Tablet not found: 104282a09c024bbaafeb5f28f91dadcb
W20250115 14:52:07.290127 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet efb8b001dd66456d9e49e9e8a6848ccf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet efb8b001dd66456d9e49e9e8a6848ccf already in progress: creating tablet
W20250115 14:52:07.290283 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 78a2d39d0296468db48b3eec374244ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a2d39d0296468db48b3eec374244ca already in progress: creating tablet
W20250115 14:52:07.290454 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f5c1f0e0b6ee4448b9ba2f78d0117197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5c1f0e0b6ee4448b9ba2f78d0117197 already in progress: creating tablet
W20250115 14:52:07.290622 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 45f124134eab4babac145404c62e7d2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45f124134eab4babac145404c62e7d2a already in progress: creating tablet
I20250115 14:52:07.291486 1162 tablet_bootstrap.cc:492] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.291586 1162 ts_tablet_manager.cc:1397] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20250115 14:52:07.292203 1162 raft_consensus.cc:357] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.292384 1162 raft_consensus.cc:383] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.292433 1162 raft_consensus.cc:738] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.292464 1164 tablet_bootstrap.cc:492] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.292583 1162 consensus_queue.cc:260] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.293036 1162 ts_tablet_manager.cc:1428] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250115 14:52:07.293224 1168 tablet_bootstrap.cc:654] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.294822 1162 tablet_bootstrap.cc:492] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:07.295372 1164 tablet_bootstrap.cc:654] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.296731 1162 tablet_bootstrap.cc:654] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.296788 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 748) took 1436 ms. Trace:
I20250115 14:52:07.296926 1167 rpcz_store.cc:276] 0115 14:52:05.859952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.860278 (+ 326us) service_pool.cc:225] Handling call
0115 14:52:07.227404 (+1367126us) ts_tablet_manager.cc:1145] Deleting tablet 171643c40c9b43ffb823f7c33789284d
0115 14:52:07.227413 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.242298 (+ 14885us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.275552 (+ 33254us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.296770 (+ 21218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1366911}
I20250115 14:52:07.298560 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_4b79de2dd7fe493f84a77ae3c477e2c8 [id=471b029f1dba4dd087deb339bba02013]). Expected version 1 got 0
I20250115 14:52:07.300974 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 750) took 1440 ms. Trace:
I20250115 14:52:07.301059 1167 rpcz_store.cc:276] 0115 14:52:05.860145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.861571 (+ 1426us) service_pool.cc:225] Handling call
0115 14:52:07.300847 (+1439276us) ts_tablet_manager.cc:1145] Deleting tablet 35301651222249d0b3439e0638f599f2
0115 14:52:07.300857 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.300947 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1439128}
I20250115 14:52:07.301357 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 751) took 1439 ms. Trace:
I20250115 14:52:07.301419 1167 rpcz_store.cc:276] 0115 14:52:05.861855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.862207 (+ 352us) service_pool.cc:225] Handling call
0115 14:52:07.301263 (+1439056us) ts_tablet_manager.cc:1145] Deleting tablet e59b36e986e146b9980c1f71599f32aa
0115 14:52:07.301272 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.301343 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438923}
I20250115 14:52:07.303206 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 752) took 1440 ms. Trace:
I20250115 14:52:07.303275 1167 rpcz_store.cc:276] 0115 14:52:05.862497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.862655 (+ 158us) service_pool.cc:225] Handling call
0115 14:52:07.303114 (+1440459us) ts_tablet_manager.cc:1145] Deleting tablet fbf5da59db79418498db09ec8ea3ac0a
0115 14:52:07.303123 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.303191 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1440328}
I20250115 14:52:07.304432 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 753) took 1394 ms. Trace:
I20250115 14:52:07.305011 1167 rpcz_store.cc:276] 0115 14:52:05.910400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.911023 (+ 623us) service_pool.cc:225] Handling call
0115 14:52:07.304343 (+1393320us) ts_tablet_manager.cc:1145] Deleting tablet 4341977f363d40eeac71b7d33d1b43ac
0115 14:52:07.304351 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.304417 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1393149}
I20250115 14:52:07.305518 2273 raft_consensus.cc:491] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.305629 2273 raft_consensus.cc:513] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.305761 1167 tablet_replica.cc:331] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:07.306138 2273 leader_election.cc:290] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.305922 1167 raft_consensus.cc:2238] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.306581 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c1602aef514dd29d5d2cffb043dbb6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:07.307346 983 leader_election.cc:343] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.307555 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c1602aef514dd29d5d2cffb043dbb6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:07.308007 984 leader_election.cc:343] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.308099 984 leader_election.cc:304] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.308315 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:07.308559 2273 raft_consensus.cc:2743] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:07.318940 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 005a4cd0e9f14e76b01afa49d946ee32 consensus state has no leader
I20250115 14:52:07.324709 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d360add9d39a48caa7b90f7d786f7bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.329277 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e6cb19641f49e599275a5a5fc272fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.329828 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_4b79de2dd7fe493f84a77ae3c477e2c8 [id=471b029f1dba4dd087deb339bba02013]). Expected version 1 got 0
I20250115 14:52:07.330165 446 catalog_manager.cc:5526] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:07.330451 1751 tablet_replica.cc:331] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:07.330621 1751 raft_consensus.cc:2238] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:07.330806 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet be2fd31f74a94787a5e8ccae3c5ef245 (table default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec [id=724c015c8f3f4aa581ac6faa84187b06]). Expected version 1 got 0
I20250115 14:52:07.330947 1751 raft_consensus.cc:2267] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.331447 1751 ts_tablet_manager.cc:1905] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.341101 2273 raft_consensus.cc:491] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.341200 886 tablet_service.cc:1467] Processing CreateTablet for tablet c26bb1836ee24d8fae9605d3c1818eef (DEFAULT_TABLE table=default.Table_d4b4b4d0c8e644acb0eaa155e8c59606 [id=314c91cf30bd4bfda8cf7b273edfc6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.341220 2273 raft_consensus.cc:513] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.341552 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c26bb1836ee24d8fae9605d3c1818eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.341804 2273 leader_election.cc:290] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.341989 748 tablet_service.cc:1514] Processing DeleteTablet for tablet d71a1ecd900942e5ae788c9a28c70a7a with delete_type TABLET_DATA_DELETED (replaced by tablet 0651be54e0b244a4bace9f47efc9b58a) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.342212 1166 tablet_replica.cc:331] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.342372 1166 raft_consensus.cc:2238] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.342353 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2fd31f74a94787a5e8ccae3c5ef245" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:07.342499 1166 raft_consensus.cc:2267] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:07.342799 983 leader_election.cc:343] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.343144 1166 ts_tablet_manager.cc:1905] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.343611 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2fd31f74a94787a5e8ccae3c5ef245" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:07.344081 984 leader_election.cc:343] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.344173 984 leader_election.cc:304] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.344530 2273 raft_consensus.cc:2743] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.344558 748 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea8faf380974ad9888b7da3fab6bf3e (DEFAULT_TABLE table=default.Table_d100b9e08d3045d083f5800a8d651450 [id=c64973aed838415d9a339344fe777e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.344862 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea8faf380974ad9888b7da3fab6bf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.344928 1019 tablet_service.cc:1467] Processing CreateTablet for tablet c26bb1836ee24d8fae9605d3c1818eef (DEFAULT_TABLE table=default.Table_d4b4b4d0c8e644acb0eaa155e8c59606 [id=314c91cf30bd4bfda8cf7b273edfc6e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.345211 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c26bb1836ee24d8fae9605d3c1818eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.345724 752 tablet_service.cc:1467] Processing CreateTablet for tablet 4911a010ccd2462fae7cb940e24f030c (DEFAULT_TABLE table=default.Table_d1c9cfaca7974878b6bf92f9b8cdbbf5 [id=01c64e3042e54a05992b06019e99221d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.346071 1013 tablet_service.cc:1467] Processing CreateTablet for tablet db57f7c3a4c14cc19d3a0624e5b1817a (DEFAULT_TABLE table=default.Table_0b05d8cc5e384fccb6f6949ae2c33c98 [id=3140693ea82a44e986d02483b387cc4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.346839 879 tablet_service.cc:1467] Processing CreateTablet for tablet db57f7c3a4c14cc19d3a0624e5b1817a (DEFAULT_TABLE table=default.Table_0b05d8cc5e384fccb6f6949ae2c33c98 [id=3140693ea82a44e986d02483b387cc4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.347160 1010 tablet_service.cc:1467] Processing CreateTablet for tablet f2993e1c95234a1db2ae98a8e08c43f8 (DEFAULT_TABLE table=default.Table_d60ab02c55f94f3590d4b43fb3c8fa0a [id=c41d4b193ab34f1084a8b0b24578f05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.347416 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db57f7c3a4c14cc19d3a0624e5b1817a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.347822 1011 tablet_service.cc:1467] Processing CreateTablet for tablet b3c31e24081d4a42aa933acbb80c6495 (DEFAULT_TABLE table=default.Table_f257047f59814e1a8458779252a782b9 [id=51d11f13250a43e8bc3b6988128d76d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.348181 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 894532f9a4d542a09eb51c2fcdd2907a (DEFAULT_TABLE table=default.Table_a75e911886994dffbae81cf1e8e94849 [id=a68990d66de442f8a3709cc02def1b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.348774 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6437873733584722979c1e9ae8b0730b (DEFAULT_TABLE table=default.Table_4dff2e869f8141e09dd757d6e47118f4 [id=43852683a5a64e1b85403395fa3d210c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.348891 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4911a010ccd2462fae7cb940e24f030c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.350747 882 tablet_service.cc:1467] Processing CreateTablet for tablet f2993e1c95234a1db2ae98a8e08c43f8 (DEFAULT_TABLE table=default.Table_d60ab02c55f94f3590d4b43fb3c8fa0a [id=c41d4b193ab34f1084a8b0b24578f05b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.351053 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2993e1c95234a1db2ae98a8e08c43f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.351275 893 tablet_service.cc:1467] Processing CreateTablet for tablet 894532f9a4d542a09eb51c2fcdd2907a (DEFAULT_TABLE table=default.Table_a75e911886994dffbae81cf1e8e94849 [id=a68990d66de442f8a3709cc02def1b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.351543 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894532f9a4d542a09eb51c2fcdd2907a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.350768 880 tablet_service.cc:1467] Processing CreateTablet for tablet b3c31e24081d4a42aa933acbb80c6495 (DEFAULT_TABLE table=default.Table_f257047f59814e1a8458779252a782b9 [id=51d11f13250a43e8bc3b6988128d76d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.351855 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c31e24081d4a42aa933acbb80c6495. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.306911 1167 raft_consensus.cc:2267] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.350993 891 tablet_service.cc:1467] Processing CreateTablet for tablet 6437873733584722979c1e9ae8b0730b (DEFAULT_TABLE table=default.Table_4dff2e869f8141e09dd757d6e47118f4 [id=43852683a5a64e1b85403395fa3d210c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.356132 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6437873733584722979c1e9ae8b0730b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.357919 1167 ts_tablet_manager.cc:1905] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.359925 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 8e6cc08065494d9e85eeff5044b44ba6 (table default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]) successfully deleted
I20250115 14:52:07.360785 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_a08bf8546b9148dc97c582984b4409c8 [id=7628dbb6818f443680a97fef665d6d2b]) successfully deleted
W20250115 14:52:07.361119 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet fbf5da59db79418498db09ec8ea3ac0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbf5da59db79418498db09ec8ea3ac0a already in progress: creating tablet
W20250115 14:52:07.361480 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 35301651222249d0b3439e0638f599f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35301651222249d0b3439e0638f599f2 already in progress: creating tablet
I20250115 14:52:07.361799 752 tablet_service.cc:1467] Processing CreateTablet for tablet 32cb8768ca424aea87682295f30830a9 (DEFAULT_TABLE table=default.Table_69711e6d76ee4d289ccb17b881ec97bf [id=fea6aa6522da441ca8e2ec5f0f9b0341]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.362463 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32cb8768ca424aea87682295f30830a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.362128 751 tablet_service.cc:1467] Processing CreateTablet for tablet 96349985df1e49e9b942eb610ccf70f9 (DEFAULT_TABLE table=default.Table_eb0ebfb0281048dbae59684c7b4e0269 [id=dc804976839545529ba04e310510bee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.362425 749 tablet_service.cc:1467] Processing CreateTablet for tablet 21c55d3525114e4492830964c3be78cc (DEFAULT_TABLE table=default.Table_cffb7afceb4a4e8abebbe51a06b58b7a [id=d0e7583a64cb40ef9e3695692a96fc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.362149 741 tablet_service.cc:1467] Processing CreateTablet for tablet 251b953705c844dda3628d5d5464d4a1 (DEFAULT_TABLE table=default.Table_1a17daf7f1be4dcd938b8a2e2f7156d1 [id=e10b7ad282a84a1b929dfe04c6480f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.362972 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c55d3525114e4492830964c3be78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.363128 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96349985df1e49e9b942eb610ccf70f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.363426 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 251b953705c844dda3628d5d5464d4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.368561 748 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd4e58cf2544d0c9bfa159d8204418b (DEFAULT_TABLE table=default.Table_e93ba6767713429194e5e07a2b1afc59 [id=01358dae9701476998f124ef53fc39ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.368898 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bd4e58cf2544d0c9bfa159d8204418b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.379276 1167 ts_tablet_manager.cc:1918] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:07.379436 1167 log.cc:1198] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/83097c5bfe364418a213236435edc40c
I20250115 14:52:07.379817 1167 ts_tablet_manager.cc:1939] T 83097c5bfe364418a213236435edc40c P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:07.380781 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 754) took 1469 ms. Trace:
I20250115 14:52:07.380919 1167 rpcz_store.cc:276] 0115 14:52:05.911376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.911783 (+ 407us) service_pool.cc:225] Handling call
0115 14:52:07.305720 (+1393937us) ts_tablet_manager.cc:1145] Deleting tablet 83097c5bfe364418a213236435edc40c
0115 14:52:07.305728 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.375180 (+ 69452us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.379205 (+ 4025us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.380765 (+ 1560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1393792}
I20250115 14:52:07.381562 1166 ts_tablet_manager.cc:1918] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.381695 1166 log.cc:1198] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d71a1ecd900942e5ae788c9a28c70a7a
W20250115 14:52:07.382009 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e59b36e986e146b9980c1f71599f32aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e59b36e986e146b9980c1f71599f32aa already in progress: creating tablet
I20250115 14:52:07.382066 1166 ts_tablet_manager.cc:1939] T d71a1ecd900942e5ae788c9a28c70a7a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.382812 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 4341977f363d40eeac71b7d33d1b43ac (table default.Table_ec25afcaa18b41b3906954d80092245d [id=b1e6efb09adf4c898362ed62f9fb98f9]) successfully deleted
I20250115 14:52:07.384541 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c31e24081d4a42aa933acbb80c6495. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.384763 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db57f7c3a4c14cc19d3a0624e5b1817a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.385233 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2993e1c95234a1db2ae98a8e08c43f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.383221 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_092faa925c9f45838cbc3a14b0fc60dd" } new_table_name: "default.Table_062a9fcca27a4d90b73742167b635ae8" modify_external_catalogs: true
I20250115 14:52:07.385128 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9a5f72bcad5e4bfe9b0e4ee3eeffd7f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:07.385601 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 894532f9a4d542a09eb51c2fcdd2907a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.385748 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a5f72bcad5e4bfe9b0e4ee3eeffd7f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.386807 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2d55a4e32448fbb98f704dbf1657b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.387512 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d4ae47e4d4274e66adc1753796b0ae1e" } new_table_name: "default.Table_958dafb4bff949ae8e00c6981747aa85" modify_external_catalogs: true
I20250115 14:52:07.388546 1162 tablet_bootstrap.cc:492] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.388687 1162 ts_tablet_manager.cc:1397] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.094s user 0.003s sys 0.000s
I20250115 14:52:07.389132 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6437873733584722979c1e9ae8b0730b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.389544 1162 raft_consensus.cc:357] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.389703 1162 raft_consensus.cc:383] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.389746 1162 raft_consensus.cc:738] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.389912 1162 consensus_queue.cc:260] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.390355 1162 ts_tablet_manager.cc:1428] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:07.390594 1162 tablet_bootstrap.cc:492] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:07.392026 1162 tablet_bootstrap.cc:654] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.392439 455 master_service.cc:946] ReplaceTablet: received request to replace tablet e3c54fdc672f4709a44b2542212dfb5e from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:07.392544 1751 ts_tablet_manager.cc:1918] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:07.392623 1751 log.cc:1198] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/83097c5bfe364418a213236435edc40c
I20250115 14:52:07.392796 1164 tablet_bootstrap.cc:492] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.392920 1751 ts_tablet_manager.cc:1939] T 83097c5bfe364418a213236435edc40c P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:07.392920 1164 ts_tablet_manager.cc:1397] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.101s user 0.004s sys 0.008s
I20250115 14:52:07.392928 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e01109831b5f43a7b10352cdb82cc440"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.393420 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e01109831b5f43a7b10352cdb82cc440 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.393512 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 780) took 1512 ms. Trace:
I20250115 14:52:07.393553 1164 raft_consensus.cc:357] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.393579 1751 rpcz_store.cc:276] 0115 14:52:05.881071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.881121 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:07.282276 (+1401155us) ts_tablet_manager.cc:1145] Deleting tablet 83097c5bfe364418a213236435edc40c
0115 14:52:07.330405 (+ 48129us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.338070 (+ 7665us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.392475 (+ 54405us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.393494 (+ 1019us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1401005}
I20250115 14:52:07.393720 1164 raft_consensus.cc:383] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.393784 1164 raft_consensus.cc:738] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.393946 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 781) took 1494 ms. Trace:
I20250115 14:52:07.393915 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_225689bca59c415d884b3d3ca93d556b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:07.393981 1164 consensus_queue.cc:260] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.394008 1751 rpcz_store.cc:276] 0115 14:52:05.899706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.903214 (+ 3508us) service_pool.cc:225] Handling call
0115 14:52:07.393816 (+1490602us) ts_tablet_manager.cc:1145] Deleting tablet 919ab6ec6f3745e082aa441dfebc235e
0115 14:52:07.393825 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.393931 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490406}
I20250115 14:52:07.394219 1751 tablet_replica.cc:331] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:07.394331 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_225689bca59c415d884b3d3ca93d556b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.394395 1751 raft_consensus.cc:2238] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.394517 1751 raft_consensus.cc:2267] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.394968 1751 ts_tablet_manager.cc:1905] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.395426 886 tablet_service.cc:1514] Processing DeleteTablet for tablet b5937d6ad0494734a402fdc594eb9415 with delete_type TABLET_DATA_DELETED (replaced by tablet f2ba4086a6bf4bfa83e740b31dfb53f8) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.395743 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_eb0ebfb0281048dbae59684c7b4e0269" } modify_external_catalogs: true
I20250115 14:52:07.395912 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_eb0ebfb0281048dbae59684c7b4e0269" } modify_external_catalogs: true
I20250115 14:52:07.396091 1164 ts_tablet_manager.cc:1428] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:07.396386 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5e402e29daa745aa81afd155e5c68bb4" } modify_external_catalogs: true
I20250115 14:52:07.396541 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5e402e29daa745aa81afd155e5c68bb4" } modify_external_catalogs: true
I20250115 14:52:07.397151 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 064f890830304758a6840a1502e59080 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.399752 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d71a1ecd900942e5ae788c9a28c70a7a (table default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]) successfully deleted
I20250115 14:52:07.400058 1843 raft_consensus.cc:491] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.400164 1843 raft_consensus.cc:513] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.400350 886 tablet_service.cc:1514] Processing DeleteTablet for tablet efb8b001dd66456d9e49e9e8a6848ccf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.400692 1843 leader_election.cc:290] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.401096 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a2d39d0296468db48b3eec374244ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.401468 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c1f0e0b6ee4448b9ba2f78d0117197 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c545ef40b4743c89b9e9165982d4b2f) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.401863 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f124134eab4babac145404c62e7d2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.401914 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 064f890830304758a6840a1502e59080 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.402073 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet efb8b001dd66456d9e49e9e8a6848ccf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.402220 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 78a2d39d0296468db48b3eec374244ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.402249 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 9853dcdfee5f43de84798cfa0bef674e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.402335 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c1f0e0b6ee4448b9ba2f78d0117197 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c545ef40b4743c89b9e9165982d4b2f) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.402596 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abde784f9c948cb96e6cc93c30817bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.402935 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 759a4ebe7b0444128ddd31a78aad463e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.403288 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 50273a0019bb4cd8977f7ef014372d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 70425c25919b4d0b999450210e1cb33f) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.403623 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 01291d41a2b345b4bfd00e6be0fb0129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.404014 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ce65604c294021af93632ed073f627 with delete_type TABLET_DATA_DELETED (replaced by tablet 339dd408dab7430c980058af3af04433) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.407052 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9853dcdfee5f43de84798cfa0bef674e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:07.408464 1164 tablet_bootstrap.cc:492] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.410446 1164 tablet_bootstrap.cc:654] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:07.410992 848 leader_election.cc:343] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:07.411144 847 leader_election.cc:343] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 9853dcdfee5f43de84798cfa0bef674e
I20250115 14:52:07.411240 847 leader_election.cc:304] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.411763 2200 raft_consensus.cc:2743] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.414207 2273 raft_consensus.cc:491] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.414314 2273 raft_consensus.cc:513] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.414806 2273 leader_election.cc:290] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.415244 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282d988bc41443c381bb0ba69b618f5d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:07.415817 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282d988bc41443c381bb0ba69b618f5d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:07.416226 983 leader_election.cc:343] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:07.416554 984 leader_election.cc:343] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.416637 984 leader_election.cc:304] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.417054 2241 raft_consensus.cc:2743] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.417701 1164 tablet_bootstrap.cc:492] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.417847 1164 ts_tablet_manager.cc:1397] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20250115 14:52:07.418567 1164 raft_consensus.cc:357] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.418772 1164 raft_consensus.cc:383] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.418845 1164 raft_consensus.cc:738] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.419056 1164 consensus_queue.cc:260] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.420281 1164 ts_tablet_manager.cc:1428] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:07.420696 1164 tablet_bootstrap.cc:492] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.422289 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07686c40324233a49e4c49aeed73f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.422847 886 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea8faf380974ad9888b7da3fab6bf3e (DEFAULT_TABLE table=default.Table_d100b9e08d3045d083f5800a8d651450 [id=c64973aed838415d9a339344fe777e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.423169 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea8faf380974ad9888b7da3fab6bf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.423804 1166 tablet_replica.cc:331] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.423946 1166 raft_consensus.cc:2238] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.424052 1166 raft_consensus.cc:2267] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.424535 1164 tablet_bootstrap.cc:654] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.425100 1166 ts_tablet_manager.cc:1905] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.427263 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet b5937d6ad0494734a402fdc594eb9415 with delete_type TABLET_DATA_DELETED (replaced by tablet f2ba4086a6bf4bfa83e740b31dfb53f8) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.427800 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2d55a4e32448fbb98f704dbf1657b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.430421 1166 ts_tablet_manager.cc:1918] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.430538 1166 log.cc:1198] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/7d07686c40324233a49e4c49aeed73f6
I20250115 14:52:07.430796 1166 ts_tablet_manager.cc:1939] T 7d07686c40324233a49e4c49aeed73f6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.434880 1843 consensus_queue.cc:1035] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:07.435802 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea8faf380974ad9888b7da3fab6bf3e (DEFAULT_TABLE table=default.Table_d100b9e08d3045d083f5800a8d651450 [id=c64973aed838415d9a339344fe777e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.436048 1168 tablet_bootstrap.cc:492] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.436146 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ea8faf380974ad9888b7da3fab6bf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.436153 1168 ts_tablet_manager.cc:1397] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.151s user 0.008s sys 0.004s
I20250115 14:52:07.436714 1168 raft_consensus.cc:357] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.436872 1168 raft_consensus.cc:383] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.436909 1168 raft_consensus.cc:738] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.437072 1168 consensus_queue.cc:260] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.437528 1168 ts_tablet_manager.cc:1428] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:07.438140 1168 tablet_bootstrap.cc:492] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.439313 1168 tablet_bootstrap.cc:654] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.442473 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 7d07686c40324233a49e4c49aeed73f6 (table default.Table_8f5c258dbdc04de79751c3a1b995e0fb [id=e71b0f487dc0428aaa03b4cc7cf95aa0]) successfully deleted
I20250115 14:52:07.443161 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 4911a010ccd2462fae7cb940e24f030c (DEFAULT_TABLE table=default.Table_d1c9cfaca7974878b6bf92f9b8cdbbf5 [id=01c64e3042e54a05992b06019e99221d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.443665 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4911a010ccd2462fae7cb940e24f030c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.444414 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4911a010ccd2462fae7cb940e24f030c (DEFAULT_TABLE table=default.Table_d1c9cfaca7974878b6bf92f9b8cdbbf5 [id=01c64e3042e54a05992b06019e99221d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.444706 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4911a010ccd2462fae7cb940e24f030c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.446658 2200 consensus_queue.cc:1035] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:07.449048 879 tablet_service.cc:1467] Processing CreateTablet for tablet 32cb8768ca424aea87682295f30830a9 (DEFAULT_TABLE table=default.Table_69711e6d76ee4d289ccb17b881ec97bf [id=fea6aa6522da441ca8e2ec5f0f9b0341]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.449345 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32cb8768ca424aea87682295f30830a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.449339 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 32cb8768ca424aea87682295f30830a9 (DEFAULT_TABLE table=default.Table_69711e6d76ee4d289ccb17b881ec97bf [id=fea6aa6522da441ca8e2ec5f0f9b0341]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.449615 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32cb8768ca424aea87682295f30830a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.455078 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd4e58cf2544d0c9bfa159d8204418b (DEFAULT_TABLE table=default.Table_e93ba6767713429194e5e07a2b1afc59 [id=01358dae9701476998f124ef53fc39ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.455372 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bd4e58cf2544d0c9bfa159d8204418b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.462172 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ea8ab6a34caa486ab929af2cac3ce957 (table default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d]). Expected version 1 got 0
I20250115 14:52:07.463539 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 45f124134eab4babac145404c62e7d2a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.465373 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd4e58cf2544d0c9bfa159d8204418b (DEFAULT_TABLE table=default.Table_e93ba6767713429194e5e07a2b1afc59 [id=01358dae9701476998f124ef53fc39ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.465663 882 tablet_service.cc:1467] Processing CreateTablet for tablet 251b953705c844dda3628d5d5464d4a1 (DEFAULT_TABLE table=default.Table_1a17daf7f1be4dcd938b8a2e2f7156d1 [id=e10b7ad282a84a1b929dfe04c6480f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.465749 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bd4e58cf2544d0c9bfa159d8204418b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.465981 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 251b953705c844dda3628d5d5464d4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.468501 1168 tablet_bootstrap.cc:492] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.468672 1168 ts_tablet_manager.cc:1397] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.004s
I20250115 14:52:07.468901 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 338c0f6f39fd4335bbcda733b28b8bf8 (table default.Table_f367ac1a82704324a73f7079165a6e45 [id=6756cde43edd4d1e8a2768c101c4aa19]). Expected version 3 got 0
I20250115 14:52:07.469506 1168 raft_consensus.cc:357] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.470324 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 251b953705c844dda3628d5d5464d4a1 (DEFAULT_TABLE table=default.Table_1a17daf7f1be4dcd938b8a2e2f7156d1 [id=e10b7ad282a84a1b929dfe04c6480f56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.470657 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 251b953705c844dda3628d5d5464d4a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.469738 1168 raft_consensus.cc:383] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.471524 1168 raft_consensus.cc:738] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.471773 1168 consensus_queue.cc:260] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.472267 1168 ts_tablet_manager.cc:1428] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250115 14:52:07.472556 1168 tablet_bootstrap.cc:492] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.473922 1168 tablet_bootstrap.cc:654] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.481031 2273 raft_consensus.cc:491] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.481180 2273 raft_consensus.cc:513] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.481472 893 tablet_service.cc:1467] Processing CreateTablet for tablet 21c55d3525114e4492830964c3be78cc (DEFAULT_TABLE table=default.Table_cffb7afceb4a4e8abebbe51a06b58b7a [id=d0e7583a64cb40ef9e3695692a96fc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.481781 2273 leader_election.cc:290] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.481947 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c55d3525114e4492830964c3be78cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.482249 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83415f1c07ad495cabd287bdcd54dad4" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:07.482364 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83415f1c07ad495cabd287bdcd54dad4" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:07.482434 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 21c55d3525114e4492830964c3be78cc (DEFAULT_TABLE table=default.Table_cffb7afceb4a4e8abebbe51a06b58b7a [id=d0e7583a64cb40ef9e3695692a96fc7b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:07.482764 983 leader_election.cc:343] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.482847 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c55d3525114e4492830964c3be78cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.482942 984 leader_election.cc:343] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.483026 984 leader_election.cc:304] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.485482 2273 raft_consensus.cc:2743] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.487532 1751 ts_tablet_manager.cc:1918] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:07.487675 1751 log.cc:1198] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d5dd41d42331494e9c1b021816d0287e
I20250115 14:52:07.488221 1751 ts_tablet_manager.cc:1939] T d5dd41d42331494e9c1b021816d0287e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:07.488583 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 783) took 1558 ms. Trace:
I20250115 14:52:07.488726 1751 rpcz_store.cc:276] 0115 14:52:05.930391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.930445 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:07.394186 (+1463741us) ts_tablet_manager.cc:1145] Deleting tablet d5dd41d42331494e9c1b021816d0287e
0115 14:52:07.394193 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.457060 (+ 62867us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.487462 (+ 30402us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.488568 (+ 1106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1463542}
I20250115 14:52:07.489548 1751 tablet_replica.cc:331] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:07.489805 1751 raft_consensus.cc:2238] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.490005 1751 raft_consensus.cc:2267] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.490366 1164 tablet_bootstrap.cc:492] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.490540 1164 ts_tablet_manager.cc:1397] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.000s
I20250115 14:52:07.490877 1751 ts_tablet_manager.cc:1905] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.491308 1164 raft_consensus.cc:357] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.491521 1164 raft_consensus.cc:383] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.491611 1164 raft_consensus.cc:738] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.491870 1164 consensus_queue.cc:260] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.492749 1164 ts_tablet_manager.cc:1428] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:07.493672 1164 tablet_bootstrap.cc:492] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.495586 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d07686c40324233a49e4c49aeed73f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:56 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.498999 1164 tablet_bootstrap.cc:654] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.500623 2200 raft_consensus.cc:491] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.500752 2200 raft_consensus.cc:513] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.501292 2200 leader_election.cc:290] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:07.502127 847 leader_election.cc:343] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 50273a0019bb4cd8977f7ef014372d85
I20250115 14:52:07.505640 893 tablet_service.cc:1467] Processing CreateTablet for tablet 96349985df1e49e9b942eb610ccf70f9 (DEFAULT_TABLE table=default.Table_eb0ebfb0281048dbae59684c7b4e0269 [id=dc804976839545529ba04e310510bee6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:07.505708 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e9febafb7e354e289cd3f864311f14cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e9febafb7e354e289cd3f864311f14cb already in progress: creating tablet
W20250115 14:52:07.505918 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 36514c5399924e98b3bc38004a33b919 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36514c5399924e98b3bc38004a33b919 already in progress: creating tablet
I20250115 14:52:07.506107 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96349985df1e49e9b942eb610ccf70f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.506234 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 96349985df1e49e9b942eb610ccf70f9 (DEFAULT_TABLE table=default.Table_eb0ebfb0281048dbae59684c7b4e0269 [id=dc804976839545529ba04e310510bee6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.506520 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96349985df1e49e9b942eb610ccf70f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.501647 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50273a0019bb4cd8977f7ef014372d85" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:07.511256 848 leader_election.cc:343] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.511397 848 leader_election.cc:304] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.511637 2200 raft_consensus.cc:2743] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.511889 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 756) took 1598 ms. Trace:
I20250115 14:52:07.511993 1167 rpcz_store.cc:276] 0115 14:52:05.913343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.913612 (+ 269us) service_pool.cc:225] Handling call
0115 14:52:07.381227 (+1467615us) ts_tablet_manager.cc:1145] Deleting tablet 919ab6ec6f3745e082aa441dfebc235e
0115 14:52:07.381236 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.511876 (+130640us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1467473}
I20250115 14:52:07.512405 1167 tablet_replica.cc:331] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:07.512459 1168 tablet_bootstrap.cc:492] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.512603 1167 raft_consensus.cc:2238] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:07.512631 1168 ts_tablet_manager.cc:1397] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.002s
I20250115 14:52:07.513497 1168 raft_consensus.cc:357] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.513708 1168 raft_consensus.cc:383] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.513820 1168 raft_consensus.cc:738] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.514056 1168 consensus_queue.cc:260] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.515971 1168 ts_tablet_manager.cc:1428] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:52:07.516243 1168 tablet_bootstrap.cc:492] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.517457 2273 raft_consensus.cc:491] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.517570 2273 raft_consensus.cc:513] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.518168 2273 leader_election.cc:290] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.518649 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aeca346a7746468dbbff053d933472" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:07.519675 1168 tablet_bootstrap.cc:654] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:07.524755 983 leader_election.cc:343] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.525223 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aeca346a7746468dbbff053d933472" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:07.525703 984 leader_election.cc:343] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.525811 984 leader_election.cc:304] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.526129 2273 raft_consensus.cc:2743] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.528954 1162 tablet_bootstrap.cc:492] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.529079 1162 ts_tablet_manager.cc:1397] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.139s user 0.000s sys 0.002s
I20250115 14:52:07.529330 1167 raft_consensus.cc:2267] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.529847 1162 raft_consensus.cc:357] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.530068 1162 raft_consensus.cc:383] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.530117 1162 raft_consensus.cc:738] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.530293 1167 ts_tablet_manager.cc:1905] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.530292 1162 consensus_queue.cc:260] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.530702 1162 ts_tablet_manager.cc:1428] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:07.530920 1162 tablet_bootstrap.cc:492] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:07.532802 1162 tablet_bootstrap.cc:654] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.510967 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d25c13a8ea78433ea4d007bbc8a846a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.534448 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d25c13a8ea78433ea4d007bbc8a846a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.512086 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_83379c7aa94244638437e340ad9da9c5" } new_table_name: "default.Table_05b7f1dc376c42c5ad11e9635a3a738d" modify_external_catalogs: true
W20250115 14:52:07.538684 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6ce498ffd3f04c5d8eb642739a838903 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ce498ffd3f04c5d8eb642739a838903 already in progress: creating tablet
I20250115 14:52:07.539376 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 171643c40c9b43ffb823f7c33789284d (table default.Table_a08bf8546b9148dc97c582984b4409c8 [id=7628dbb6818f443680a97fef665d6d2b]) successfully deleted
W20250115 14:52:07.539747 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 35301651222249d0b3439e0638f599f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35301651222249d0b3439e0638f599f2 already in progress: creating tablet
W20250115 14:52:07.539968 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e59b36e986e146b9980c1f71599f32aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e59b36e986e146b9980c1f71599f32aa already in progress: creating tablet
W20250115 14:52:07.540191 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet fbf5da59db79418498db09ec8ea3ac0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fbf5da59db79418498db09ec8ea3ac0a already in progress: creating tablet
W20250115 14:52:07.540417 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 4341977f363d40eeac71b7d33d1b43ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4341977f363d40eeac71b7d33d1b43ac already in progress: creating tablet
I20250115 14:52:07.540923 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 83097c5bfe364418a213236435edc40c (table default.Table_81803594c5c741619ae708256092f827 [id=caa6adb7563c41088e5120993c46c54f]) successfully deleted
I20250115 14:52:07.540824 2234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef770dae727b42f5a9513b6f3d68b84b. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:52:07.541100 2234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [sys.catalog]: This master's current role is: FOLLOWER
I20250115 14:52:07.541253 2290 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef770dae727b42f5a9513b6f3d68b84b)
I20250115 14:52:07.541457 2290 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:52:07.542320 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet fa57c1fa96594a40b2551ad503127e4f with delete_type TABLET_DATA_DELETED (replaced by tablet 9c35882d2e5c464c9cfad496ba9d6af9) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.544262 2290 leader_election.cc:290] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef770dae727b42f5a9513b6f3d68b84b (127.0.104.62:35583), 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227)
W20250115 14:52:07.513089 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7d07686c40324233a49e4c49aeed73f6 because the tablet was not found. No further retry: Not found: Tablet not found: 7d07686c40324233a49e4c49aeed73f6
I20250115 14:52:07.548508 2273 raft_consensus.cc:491] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.548619 2273 raft_consensus.cc:513] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.549161 2273 leader_election.cc:290] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.549635 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a11b7b0dd6494daf7b017edd4317a5" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:07.550212 983 leader_election.cc:343] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.550588 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a11b7b0dd6494daf7b017edd4317a5" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:07.551038 984 leader_election.cc:343] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.551124 984 leader_election.cc:304] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.551429 2273 raft_consensus.cc:2743] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.551813 1958 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43174 (request call id 0) took 4131 ms. Trace:
I20250115 14:52:07.552944 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_57cdcb46b45d430b84affd591c061ced" } modify_external_catalogs: true
I20250115 14:52:07.553138 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_57cdcb46b45d430b84affd591c061ced" } modify_external_catalogs: true
I20250115 14:52:07.554636 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f1bd3a4a8eb5403ca64cf59d41757e64" } modify_external_catalogs: true
I20250115 14:52:07.554787 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f1bd3a4a8eb5403ca64cf59d41757e64" } modify_external_catalogs: true
I20250115 14:52:07.555428 1164 tablet_bootstrap.cc:492] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.555541 1164 ts_tablet_manager.cc:1397] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.000s
W20250115 14:52:07.555919 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9727430a15574ec0b24a5a64da744cac consensus state has no leader
I20250115 14:52:07.556226 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_af26e85d3ff84c6296a22baee6440af2" } new_table_name: "default.Table_b053773cca094e6499791ad922f1fe10" modify_external_catalogs: true
I20250115 14:52:07.556213 1164 raft_consensus.cc:357] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.556404 1164 raft_consensus.cc:383] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.556459 1164 raft_consensus.cc:738] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.558048 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e3c54fdc672f4709a44b2542212dfb5e
I20250115 14:52:07.558441 455 catalog_manager.cc:6596] ReplaceTablet: tablet e3c54fdc672f4709a44b2542212dfb5e deleted and replaced by tablet 2087b4ce466f4167b7c32c7074dcf6e7
I20250115 14:52:07.558578 1164 consensus_queue.cc:260] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.558799 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cb164ba6d8f74120ae308a71c1a936db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:07.559059 1164 ts_tablet_manager.cc:1428] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250115 14:52:07.559262 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb164ba6d8f74120ae308a71c1a936db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.560138 1164 tablet_bootstrap.cc:492] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.560536 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b3a6b57353484cdf8f2dfde2738ae0c2" } modify_external_catalogs: true
I20250115 14:52:07.560693 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b3a6b57353484cdf8f2dfde2738ae0c2" } modify_external_catalogs: true
I20250115 14:52:07.561141 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 96349985df1e49e9b942eb610ccf70f9
I20250115 14:52:07.561712 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_18716bd80651469abf46d1c1e1a8729a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.562170 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18716bd80651469abf46d1c1e1a8729a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.563580 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 80c75a77d7154b5cb05a5d9ac8a20787
I20250115 14:52:07.564188 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_298f14ed9b9b41fc802a48c4f1bbe07f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.564638 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_298f14ed9b9b41fc802a48c4f1bbe07f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.564831 2308 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250115 14:52:07.568346 2225 raft_consensus.cc:491] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.568466 2225 raft_consensus.cc:513] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.568590 1164 tablet_bootstrap.cc:654] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.569221 2225 leader_election.cc:290] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.569491 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e94ea76507f4646909ac0c1b2c3136a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:07.569880 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e94ea76507f4646909ac0c1b2c3136a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:07.570653 710 leader_election.cc:304] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:07.570873 2225 raft_consensus.cc:2743] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.572925 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1]). Expected version 3 got 0
I20250115 14:52:07.573109 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:07.575548 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ea8ab6a34caa486ab929af2cac3ce957 (table default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d]). Expected version 1 got 0
I20250115 14:52:07.575774 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1]). Expected version 3 got 0
I20250115 14:52:07.584185 2312 tablet.cc:1721] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:07.584705 1162 tablet_bootstrap.cc:492] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.584826 1162 ts_tablet_manager.cc:1397] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20250115 14:52:07.585445 1162 raft_consensus.cc:357] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.585613 1162 raft_consensus.cc:383] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.585656 1162 raft_consensus.cc:738] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.585873 1162 consensus_queue.cc:260] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.586365 1162 ts_tablet_manager.cc:1428] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250115 14:52:07.586661 1162 tablet_bootstrap.cc:492] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:07.588950 1168 tablet_bootstrap.cc:492] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.589124 1168 ts_tablet_manager.cc:1397] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.073s user 0.003s sys 0.000s
I20250115 14:52:07.589200 1164 tablet_bootstrap.cc:492] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.589378 1164 ts_tablet_manager.cc:1397] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250115 14:52:07.589874 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e95d09efd2e44cc81cdcbf87ee453fe with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd94e4a7fef44f5be462e7a77e8a123) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.590178 1164 raft_consensus.cc:357] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.590413 1164 raft_consensus.cc:383] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.590508 1164 raft_consensus.cc:738] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.590775 1164 consensus_queue.cc:260] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.591305 1164 ts_tablet_manager.cc:1428] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:07.592680 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet a600fa0f0ec049dca2803bf7d2c74a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.594314 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a11b7b0dd6494daf7b017edd4317a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.594942 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 83097c5bfe364418a213236435edc40c (table default.Table_81803594c5c741619ae708256092f827 [id=caa6adb7563c41088e5120993c46c54f]) successfully deleted
I20250115 14:52:07.597276 1164 tablet_bootstrap.cc:492] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.598814 1164 tablet_bootstrap.cc:654] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.601263 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]) successfully deleted
W20250115 14:52:07.601488 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet abccc613e1754839a4c7a8f5c448a852 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abccc613e1754839a4c7a8f5c448a852 already in progress: creating tablet
I20250115 14:52:07.603624 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 96349985df1e49e9b942eb610ccf70f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.603825 740 tablet_service.cc:1514] Processing DeleteTablet for tablet e3c54fdc672f4709a44b2542212dfb5e with delete_type TABLET_DATA_DELETED (replaced by tablet 2087b4ce466f4167b7c32c7074dcf6e7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.603968 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c75a77d7154b5cb05a5d9ac8a20787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.604512 1166 tablet_replica.cc:331] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.604674 1166 raft_consensus.cc:2238] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.604831 1166 raft_consensus.cc:2267] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.604812 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4b79de2dd7fe493f84a77ae3c477e2c8" } new_table_name: "default.Table_ebfbdc698c8c448ca2e60757b1fd574c" modify_external_catalogs: true
I20250115 14:52:07.605304 1166 ts_tablet_manager.cc:1905] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.608630 1162 tablet_bootstrap.cc:654] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.602300 882 tablet_service.cc:1514] Processing DeleteTablet for tablet e3c54fdc672f4709a44b2542212dfb5e with delete_type TABLET_DATA_DELETED (replaced by tablet 2087b4ce466f4167b7c32c7074dcf6e7) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:07.609822 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 96349985df1e49e9b942eb610ccf70f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96349985df1e49e9b942eb610ccf70f9 already in progress: creating tablet
W20250115 14:52:07.610211 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e3c54fdc672f4709a44b2542212dfb5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3c54fdc672f4709a44b2542212dfb5e already in progress: creating tablet
I20250115 14:52:07.610687 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet e3c54fdc672f4709a44b2542212dfb5e with delete_type TABLET_DATA_DELETED (replaced by tablet 2087b4ce466f4167b7c32c7074dcf6e7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.616189 1167 ts_tablet_manager.cc:1918] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:07.618932 1166 ts_tablet_manager.cc:1918] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:07.619230 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 96349985df1e49e9b942eb610ccf70f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.621378 1166 log.cc:1198] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/80c75a77d7154b5cb05a5d9ac8a20787
I20250115 14:52:07.621883 1166 ts_tablet_manager.cc:1939] T 80c75a77d7154b5cb05a5d9ac8a20787 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.622793 1751 ts_tablet_manager.cc:1918] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.622895 1751 log.cc:1198] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3a0d97fa57484b9d8cb879fedda61612
I20250115 14:52:07.623502 1164 tablet_bootstrap.cc:492] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.623659 1164 ts_tablet_manager.cc:1397] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250115 14:52:07.623154 1751 ts_tablet_manager.cc:1939] T 3a0d97fa57484b9d8cb879fedda61612 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:07.624470 1164 raft_consensus.cc:357] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.624702 1164 raft_consensus.cc:383] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.624827 1164 raft_consensus.cc:738] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.625099 1164 consensus_queue.cc:260] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:07.625715 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64994f0d3fdd43e2935786bf11544fb6 consensus state has no leader
I20250115 14:52:07.627048 1164 ts_tablet_manager.cc:1428] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:52:07.627276 1164 tablet_bootstrap.cc:492] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.627804 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_044eb9404b5446298ecd6e8f48215b59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:07.628726 1164 tablet_bootstrap.cc:654] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:07.629705 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_044eb9404b5446298ecd6e8f48215b59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:07.630009 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 18f23815196047628d562da7312b010f consensus state has no leader
I20250115 14:52:07.630093 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ba21cf54bc074d95b456fd70a9286001
I20250115 14:52:07.631369 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_043c9ff0bc444163a82cfc7c4d02c35f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.631789 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_043c9ff0bc444163a82cfc7c4d02c35f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.631906 2313 tablet.cc:1721] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:07.632555 2318 tablet.cc:1721] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:07.633589 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b0f33b5c22ee4f8bb3ee2d20c188ade0
I20250115 14:52:07.634138 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_888f5c5df23b4835a34d43300e9f3806"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.634521 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_888f5c5df23b4835a34d43300e9f3806 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.637449 1932 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" instance_seqno: 1736952710571245) as {username='slave'} at 127.0.104.3:38367; Asking this server to re-register.
I20250115 14:52:07.638922 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e924821d12045f6bc658461675aead4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.639225 1932 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" instance_seqno: 1736952709640711) as {username='slave'} at 127.0.104.1:39913; Asking this server to re-register.
I20250115 14:52:07.639426 749 tablet_service.cc:1514] Processing DeleteTablet for tablet caa1827366a34f039fd12deb855f07b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.639688 2225 raft_consensus.cc:491] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.639799 2225 raft_consensus.cc:513] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.640164 749 tablet_service.cc:1514] Processing DeleteTablet for tablet ba21cf54bc074d95b456fd70a9286001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.640230 823 heartbeater.cc:463] Registering TS with master...
I20250115 14:52:07.640354 2225 leader_election.cc:290] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.640954 749 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.641314 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]) successfully deleted
I20250115 14:52:07.642091 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2ee5dfdef14ec092a2f9d4bfd295a6" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:07.647258 1932 ts_manager.cc:194] Registered new tserver with Master: d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127)
I20250115 14:52:07.651814 769 raft_consensus.cc:3054] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:07.657378 1096 heartbeater.cc:463] Registering TS with master...
I20250115 14:52:07.658118 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 96349985df1e49e9b942eb610ccf70f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.658576 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c75a77d7154b5cb05a5d9ac8a20787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.658869 2200 raft_consensus.cc:491] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.659055 2200 raft_consensus.cc:513] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.659371 879 tablet_service.cc:1514] Processing DeleteTablet for tablet ba21cf54bc074d95b456fd70a9286001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.659431 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet ba21cf54bc074d95b456fd70a9286001 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.659611 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.659821 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 80c75a77d7154b5cb05a5d9ac8a20787 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.661428 1166 tablet_replica.cc:331] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.661619 1166 raft_consensus.cc:2238] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.661752 1166 raft_consensus.cc:2267] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.662297 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.662393 1166 ts_tablet_manager.cc:1905] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.663784 2138 consensus_queue.cc:1035] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:07.664108 1162 tablet_bootstrap.cc:492] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.664215 1162 ts_tablet_manager.cc:1397] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.000s
I20250115 14:52:07.664880 1162 raft_consensus.cc:357] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.664927 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c2ee5dfdef14ec092a2f9d4bfd295a6" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:07.665036 1162 raft_consensus.cc:383] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.665086 1162 raft_consensus.cc:738] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.665385 1162 consensus_queue.cc:260] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.665753 710 leader_election.cc:304] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:07.665880 1162 ts_tablet_manager.cc:1428] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:07.667129 2200 leader_election.cc:290] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.667356 2325 tablet.cc:1721] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:07.667255 2322 raft_consensus.cc:2743] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.667667 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8ab6a34caa486ab929af2cac3ce957" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:07.667850 1049 raft_consensus.cc:2463] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:07.668210 847 leader_election.cc:304] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:07.668502 2200 raft_consensus.cc:2798] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:07.668582 2200 raft_consensus.cc:491] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:07.668653 2200 raft_consensus.cc:3054] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:07.670248 1932 ts_manager.cc:194] Registered new tserver with Master: c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.670351 2138 consensus_queue.cc:1035] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:07.676627 1167 log.cc:1198] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d5dd41d42331494e9c1b021816d0287e
I20250115 14:52:07.677052 1167 ts_tablet_manager.cc:1939] T d5dd41d42331494e9c1b021816d0287e P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:07.678289 1751 tablet_replica.cc:331] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:07.678511 1751 raft_consensus.cc:2238] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.678699 1751 raft_consensus.cc:2267] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.679268 1751 ts_tablet_manager.cc:1905] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.683030 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3a0d97fa57484b9d8cb879fedda61612 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]) successfully deleted
W20250115 14:52:07.683272 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c14836b97c3247b39ff9072c12e642e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c14836b97c3247b39ff9072c12e642e0 already in progress: creating tablet
W20250115 14:52:07.683482 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f60e68b411364ada9b89879012d862ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f60e68b411364ada9b89879012d862ea already in progress: creating tablet
I20250115 14:52:07.684557 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8ab6a34caa486ab929af2cac3ce957" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:07.685191 848 leader_election.cc:343] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.686983 2330 tablet.cc:1721] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:07.689882 452 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:07.690930 1166 ts_tablet_manager.cc:1918] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.691022 1166 log.cc:1198] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/0e924821d12045f6bc658461675aead4
I20250115 14:52:07.691267 1166 ts_tablet_manager.cc:1939] T 0e924821d12045f6bc658461675aead4 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.691854 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.697018 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_c3e4c1d6466242fb8b2b5d97d504836b [id=be99ec07535243a68c1b3ee1be0c62e7] alter complete (version 3)
I20250115 14:52:07.705237 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 8f8efcf1a2db44469cd8dda0b59c34ba
I20250115 14:52:07.707314 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8efcf1a2db44469cd8dda0b59c34ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.707762 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 759) took 1729 ms. Trace:
I20250115 14:52:07.707896 1167 rpcz_store.cc:276] 0115 14:52:05.977818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:05.977857 (+ 39us) service_pool.cc:225] Handling call
0115 14:52:07.512368 (+1534511us) ts_tablet_manager.cc:1145] Deleting tablet d5dd41d42331494e9c1b021816d0287e
0115 14:52:07.512377 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.613242 (+100865us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.616116 (+ 2874us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.707746 (+ 91630us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1534337}
I20250115 14:52:07.708431 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8efcf1a2db44469cd8dda0b59c34ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.708449 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8efcf1a2db44469cd8dda0b59c34ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.708951 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d5dd41d42331494e9c1b021816d0287e (table default.Table_6db88245ecc54e498a261791c7cc2103 [id=1ab58cc2377b41589f5df2505f6b1208]) successfully deleted
I20250115 14:52:07.709268 1167 tablet_replica.cc:331] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:07.709483 1167 raft_consensus.cc:2238] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.709668 1167 raft_consensus.cc:2267] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.710203 1167 ts_tablet_manager.cc:1905] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.712116 2326 tablet.cc:1721] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:07.714260 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_062a9fcca27a4d90b73742167b635ae8 [id=5d05be5a21a44fe0b75c284bf3c83625] alter complete (version 3)
I20250115 14:52:07.711023 2312 tablet.cc:1721] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:07.715440 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 2484885df0da46f381d625a26e211c1e (table default.Table_8652d6d9ec084553b5183aa372c2554a [id=c29c4105b83d4df9b617997c0c9c5b14]). Expected version 1 got 0
I20250115 14:52:07.715605 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 92aeca346a7746468dbbff053d933472 (table default.Table_b3e4daa02d5e4d8ca7f0710e6c5f8260 [id=60ff7f730d394990a0263e322d28d7ff]). Expected version 3 got 0
I20250115 14:52:07.715745 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1]). Expected version 3 got 0
I20250115 14:52:07.717218 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 83415f1c07ad495cabd287bdcd54dad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.717543 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 377d711f052545ddbe499afe7ba3de28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.719270 749 tablet_service.cc:1467] Processing CreateTablet for tablet 578d2d27c18a491888fb67131f82f73e (DEFAULT_TABLE table=default.Table_236720cf03dd4cf7b5e1943e11dbe3d8 [id=21570890a8b84fec872723babc058d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.719627 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578d2d27c18a491888fb67131f82f73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.719803 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:07.717247 1932 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "73c86158cc2641f182ea50a06634592d" instance_seqno: 1736952710099038) as {username='slave'} at 127.0.104.2:47691; Asking this server to re-register.
I20250115 14:52:07.725762 1018 tablet_service.cc:1467] Processing CreateTablet for tablet ffa9cd1279fe4e0c9168472993a983dd (DEFAULT_TABLE table=default.Table_7f957146690846bd9bf51f27431bd54d [id=8aaa25c09c714ab892df7903002de50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.726125 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa9cd1279fe4e0c9168472993a983dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.726717 740 tablet_service.cc:1467] Processing CreateTablet for tablet c1120fa74c3648b0aaeab7209b8ac7ba (DEFAULT_TABLE table=default.Table_b7f7672a657044a38068152b29444a85 [id=2fb1232f22914ba88022cd7cffd8cd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.727149 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1120fa74c3648b0aaeab7209b8ac7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.725762 1015 tablet_service.cc:1467] Processing CreateTablet for tablet c1120fa74c3648b0aaeab7209b8ac7ba (DEFAULT_TABLE table=default.Table_b7f7672a657044a38068152b29444a85 [id=2fb1232f22914ba88022cd7cffd8cd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.728893 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1120fa74c3648b0aaeab7209b8ac7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.732158 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 760175a3a06c41bdad5889b12dd32655 (DEFAULT_TABLE table=default.Table_1692c29b37a04b5697933d4456c5319e [id=7da2b1d9ad6847d28af445b1fb7847f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.732494 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760175a3a06c41bdad5889b12dd32655. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.734910 740 tablet_service.cc:1467] Processing CreateTablet for tablet ffa9cd1279fe4e0c9168472993a983dd (DEFAULT_TABLE table=default.Table_7f957146690846bd9bf51f27431bd54d [id=8aaa25c09c714ab892df7903002de50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.735200 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa9cd1279fe4e0c9168472993a983dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.736538 752 tablet_service.cc:1467] Processing CreateTablet for tablet 760175a3a06c41bdad5889b12dd32655 (DEFAULT_TABLE table=default.Table_1692c29b37a04b5697933d4456c5319e [id=7da2b1d9ad6847d28af445b1fb7847f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.736805 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760175a3a06c41bdad5889b12dd32655. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.713615 1168 raft_consensus.cc:357] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.725862 960 heartbeater.cc:463] Registering TS with master...
I20250115 14:52:07.744792 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 0e924821d12045f6bc658461675aead4 (table default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]) successfully deleted
I20250115 14:52:07.745117 1162 tablet_bootstrap.cc:492] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:07.746347 1162 tablet_bootstrap.cc:654] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.747771 1164 tablet_bootstrap.cc:492] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.749330 1164 ts_tablet_manager.cc:1397] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.122s user 0.011s sys 0.007s
I20250115 14:52:07.753086 1164 raft_consensus.cc:357] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.754102 1436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 755 } } ignore_live_leader: false dest_uuid: "1cc2e23c560f4096be7b14006bdfc525" is_pre_election: true
I20250115 14:52:07.754870 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 755 } } ignore_live_leader: false dest_uuid: "ef770dae727b42f5a9513b6f3d68b84b" is_pre_election: true
I20250115 14:52:07.748883 1166 tablet_replica.cc:331] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.756413 1164 raft_consensus.cc:383] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.756748 1897 leader_election.cc:304] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d97cf5f1fbb4bc6883711020e2fa3f3; no voters: 1cc2e23c560f4096be7b14006bdfc525, ef770dae727b42f5a9513b6f3d68b84b
I20250115 14:52:07.757125 1166 raft_consensus.cc:2238] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.757189 2290 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 14:52:07.757330 1166 raft_consensus.cc:2267] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.757422 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1]). Expected version 3 got 0
I20250115 14:52:07.757711 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:07.757827 1164 raft_consensus.cc:738] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.758209 1166 ts_tablet_manager.cc:1905] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.758188 1164 consensus_queue.cc:260] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.758639 1164 ts_tablet_manager.cc:1428] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250115 14:52:07.759845 1168 raft_consensus.cc:383] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.759948 1168 raft_consensus.cc:738] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.759860 2236 raft_consensus.cc:491] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.760205 2236 raft_consensus.cc:513] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.761051 2236 leader_election.cc:290] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.763339 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 6942d102069d4c33b23fa2c0b29f94a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.763530 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 95718d39a7824517a36c49c3b86fef30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.764225 882 tablet_service.cc:1467] Processing CreateTablet for tablet 578d2d27c18a491888fb67131f82f73e (DEFAULT_TABLE table=default.Table_236720cf03dd4cf7b5e1943e11dbe3d8 [id=21570890a8b84fec872723babc058d00]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:07.761838 847 leader_election.cc:343] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 759a4ebe7b0444128ddd31a78aad463e
I20250115 14:52:07.765695 1164 tablet_bootstrap.cc:492] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.766642 1168 consensus_queue.cc:260] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.770587 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:07.771226 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d8c53fdf16243f0a614e48ec4ac94f4 with delete_type TABLET_DATA_DELETED (replaced by tablet 3b82924e735846b2914520684d1b8fc7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.778990 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_ebfbdc698c8c448ca2e60757b1fd574c [id=471b029f1dba4dd087deb339bba02013] alter complete (version 2)
I20250115 14:52:07.781642 1166 ts_tablet_manager.cc:1918] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.781765 1166 log.cc:1198] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/caa1827366a34f039fd12deb855f07b6
I20250115 14:52:07.782079 1166 ts_tablet_manager.cc:1939] T caa1827366a34f039fd12deb855f07b6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.782474 1164 tablet_bootstrap.cc:654] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.783000 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet caa1827366a34f039fd12deb855f07b6 (table default.Table_a43e4255e5c3414aa47bff8bbe111d13 [id=aa88c2d281fe4856b6d6af91c9359f23]) successfully deleted
I20250115 14:52:07.783664 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 578d2d27c18a491888fb67131f82f73e (DEFAULT_TABLE table=default.Table_236720cf03dd4cf7b5e1943e11dbe3d8 [id=21570890a8b84fec872723babc058d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.783715 740 tablet_service.cc:1467] Processing CreateTablet for tablet ab6c8274e093404fa5126565e6de73f8 (DEFAULT_TABLE table=default.Table_df72eb78efd74af192eb92233075998c [id=ea79ed81e0f64f749eed60c60830801f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.784057 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6c8274e093404fa5126565e6de73f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.785090 1015 tablet_service.cc:1467] Processing CreateTablet for tablet ab6c8274e093404fa5126565e6de73f8 (DEFAULT_TABLE table=default.Table_df72eb78efd74af192eb92233075998c [id=ea79ed81e0f64f749eed60c60830801f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.785441 886 tablet_service.cc:1467] Processing CreateTablet for tablet ffa9cd1279fe4e0c9168472993a983dd (DEFAULT_TABLE table=default.Table_7f957146690846bd9bf51f27431bd54d [id=8aaa25c09c714ab892df7903002de50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.785645 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6c8274e093404fa5126565e6de73f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.788789 893 tablet_service.cc:1467] Processing CreateTablet for tablet c1120fa74c3648b0aaeab7209b8ac7ba (DEFAULT_TABLE table=default.Table_b7f7672a657044a38068152b29444a85 [id=2fb1232f22914ba88022cd7cffd8cd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.791260 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578d2d27c18a491888fb67131f82f73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.792227 752 tablet_service.cc:1467] Processing CreateTablet for tablet 3bef17ce7b43457bb11dba0bb1fd70dd (DEFAULT_TABLE table=default.Table_3950c52dccfb443bbfd1bed7a0007bd7 [id=6a7b69f7dd9a4b799eff42705854708c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.792595 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bef17ce7b43457bb11dba0bb1fd70dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.794030 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ba21cf54bc074d95b456fd70a9286001 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba21cf54bc074d95b456fd70a9286001 already in progress: creating tablet
W20250115 14:52:07.794613 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 already in progress: creating tablet
W20250115 14:52:07.795223 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 8f8efcf1a2db44469cd8dda0b59c34ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f8efcf1a2db44469cd8dda0b59c34ba already in progress: creating tablet
I20250115 14:52:07.795469 1166 tablet_replica.cc:331] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.795665 1166 raft_consensus.cc:2238] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.802090 1166 raft_consensus.cc:2267] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.806854 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_52bf98f0d5b44013ba19e15f0fc24cce" } modify_external_catalogs: true
I20250115 14:52:07.807040 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_52bf98f0d5b44013ba19e15f0fc24cce" } modify_external_catalogs: true
I20250115 14:52:07.808241 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_02e5510e4edc458bab64cbb11e942275" } modify_external_catalogs: true
I20250115 14:52:07.808439 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_02e5510e4edc458bab64cbb11e942275" } modify_external_catalogs: true
I20250115 14:52:07.809567 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f6993d1e422647d59233c375c8894af3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.810057 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6993d1e422647d59233c375c8894af3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.815229 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f72ab5996cfd4471996f0b1034d3e422"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.815708 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f72ab5996cfd4471996f0b1034d3e422 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.817405 1166 ts_tablet_manager.cc:1905] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.817912 1751 ts_tablet_manager.cc:1918] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.818024 1751 log.cc:1198] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/8442dc919d5f4d63b9b2ce7276cc0bc5
I20250115 14:52:07.818279 1751 ts_tablet_manager.cc:1939] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:07.818957 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 821) took 1074 ms. Trace:
I20250115 14:52:07.819047 1751 rpcz_store.cc:276] 0115 14:52:06.744699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:06.744753 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:07.678247 (+933494us) ts_tablet_manager.cc:1145] Deleting tablet 8442dc919d5f4d63b9b2ce7276cc0bc5
0115 14:52:07.678255 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.799984 (+121729us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.817806 (+ 17822us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.818937 (+ 1131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":933302}
I20250115 14:52:07.820073 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c5f3bc4addaa42d7acaf157e65707baa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.821133 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5f3bc4addaa42d7acaf157e65707baa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.825281 879 tablet_service.cc:1467] Processing CreateTablet for tablet 760175a3a06c41bdad5889b12dd32655 (DEFAULT_TABLE table=default.Table_1692c29b37a04b5697933d4456c5319e [id=7da2b1d9ad6847d28af445b1fb7847f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.825733 1166 ts_tablet_manager.cc:1918] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.825911 1166 log.cc:1198] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9d8c53fdf16243f0a614e48ec4ac94f4
I20250115 14:52:07.826602 1166 ts_tablet_manager.cc:1939] T 9d8c53fdf16243f0a614e48ec4ac94f4 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.827306 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 325cd3fd8b4d447998822163e575e758
I20250115 14:52:07.827585 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9d8c53fdf16243f0a614e48ec4ac94f4 (table default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]) successfully deleted
I20250115 14:52:07.828182 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 325cd3fd8b4d447998822163e575e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:07.828773 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 325cd3fd8b4d447998822163e575e758 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 325cd3fd8b4d447998822163e575e758 already in progress: creating tablet
I20250115 14:52:07.830219 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578d2d27c18a491888fb67131f82f73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.830476 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1120fa74c3648b0aaeab7209b8ac7ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.831259 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 760175a3a06c41bdad5889b12dd32655. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.831446 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c042a4f4f9bc46afb0b94df8ed74a8cf
I20250115 14:52:07.833009 740 tablet_service.cc:1514] Processing DeleteTablet for tablet c042a4f4f9bc46afb0b94df8ed74a8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.833321 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa9cd1279fe4e0c9168472993a983dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.833490 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c042a4f4f9bc46afb0b94df8ed74a8cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c042a4f4f9bc46afb0b94df8ed74a8cf already in progress: creating tablet
I20250115 14:52:07.835616 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 325cd3fd8b4d447998822163e575e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.835886 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet c042a4f4f9bc46afb0b94df8ed74a8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:07.798323 1162 tablet_bootstrap.cc:492] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.836922 2200 raft_consensus.cc:513] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.837265 1162 ts_tablet_manager.cc:1397] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.092s user 0.007s sys 0.000s
I20250115 14:52:07.838222 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e034d7761a5247c79414b7112a1c97b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.838781 1162 raft_consensus.cc:357] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.838949 1162 raft_consensus.cc:383] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.839010 1162 raft_consensus.cc:738] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.839017 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 325cd3fd8b4d447998822163e575e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.839062 891 tablet_service.cc:1467] Processing CreateTablet for tablet ab6c8274e093404fa5126565e6de73f8 (DEFAULT_TABLE table=default.Table_df72eb78efd74af192eb92233075998c [id=ea79ed81e0f64f749eed60c60830801f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.839193 1162 consensus_queue.cc:260] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.839355 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6c8274e093404fa5126565e6de73f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.839883 885 tablet_service.cc:1514] Processing DeleteTablet for tablet c042a4f4f9bc46afb0b94df8ed74a8cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:07.840087 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "759a4ebe7b0444128ddd31a78aad463e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:07.842154 1162 ts_tablet_manager.cc:1428] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:07.843288 1162 tablet_bootstrap.cc:492] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:07.843524 848 leader_election.cc:343] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.843632 848 leader_election.cc:304] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.844563 1162 tablet_bootstrap.cc:654] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.847287 1164 tablet_bootstrap.cc:492] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.847415 1164 ts_tablet_manager.cc:1397] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.002s
I20250115 14:52:07.847733 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 3bef17ce7b43457bb11dba0bb1fd70dd (DEFAULT_TABLE table=default.Table_3950c52dccfb443bbfd1bed7a0007bd7 [id=6a7b69f7dd9a4b799eff42705854708c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.848065 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bef17ce7b43457bb11dba0bb1fd70dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.848102 1164 raft_consensus.cc:357] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.848268 1164 raft_consensus.cc:383] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.848333 1164 raft_consensus.cc:738] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.848539 1164 consensus_queue.cc:260] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.849020 1164 ts_tablet_manager.cc:1428] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:07.854112 2241 raft_consensus.cc:491] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.854223 2241 raft_consensus.cc:513] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.854768 2241 leader_election.cc:290] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.855326 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2484885df0da46f381d625a26e211c1e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:07.856153 983 leader_election.cc:343] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.856618 2322 raft_consensus.cc:491] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.856695 740 tablet_service.cc:1467] Processing CreateTablet for tablet 83edbad0640c4bf39f4497522c4135e4 (DEFAULT_TABLE table=default.Table_86158e4855654f77bfa28259f0d0e38b [id=c3903ce80530492590797926806ec07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.856742 2322 raft_consensus.cc:513] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.857318 2322 leader_election.cc:290] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.858013 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 83edbad0640c4bf39f4497522c4135e4 (DEFAULT_TABLE table=default.Table_86158e4855654f77bfa28259f0d0e38b [id=c3903ce80530492590797926806ec07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.858366 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83edbad0640c4bf39f4497522c4135e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.864210 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2484885df0da46f381d625a26e211c1e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:07.864801 1164 tablet_bootstrap.cc:492] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:07.864785 984 leader_election.cc:343] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.864882 984 leader_election.cc:304] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.865199 2241 raft_consensus.cc:2743] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.865542 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83edbad0640c4bf39f4497522c4135e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.866292 1164 tablet_bootstrap.cc:654] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.867568 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd405050a2548e1b9ea7d7bf34a35af (DEFAULT_TABLE table=default.Table_29a289769ec64b96a7352fa0412f999b [id=292730d3a3c5477bae35f28e717417a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.867883 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd405050a2548e1b9ea7d7bf34a35af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.868517 749 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd405050a2548e1b9ea7d7bf34a35af (DEFAULT_TABLE table=default.Table_29a289769ec64b96a7352fa0412f999b [id=292730d3a3c5477bae35f28e717417a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.868831 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd405050a2548e1b9ea7d7bf34a35af. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.874178 710 leader_election.cc:343] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 9853dcdfee5f43de84798cfa0bef674e
I20250115 14:52:07.875061 748 tablet_service.cc:1467] Processing CreateTablet for tablet 8c545ef40b4743c89b9e9165982d4b2f (DEFAULT_TABLE table=default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.875360 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c545ef40b4743c89b9e9165982d4b2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.875756 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 109810us
I20250115 14:52:07.880040 2241 raft_consensus.cc:491] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.880154 2241 raft_consensus.cc:513] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.880868 2241 leader_election.cc:290] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:07.881424 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6942d102069d4c33b23fa2c0b29f94a6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:07.882261 983 leader_election.cc:343] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.882483 2241 consensus_queue.cc:1035] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:07.882588 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9a47006d838d44b6ae411fddec170586"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.883162 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a47006d838d44b6ae411fddec170586 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.884703 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6942d102069d4c33b23fa2c0b29f94a6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:07.885133 984 leader_election.cc:343] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:07.885236 984 leader_election.cc:304] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:07.885586 2241 raft_consensus.cc:2743] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.885875 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a75e911886994dffbae81cf1e8e94849" } new_table_name: "default.Table_06f9926d94994f899b4cb94f5a89f817" modify_external_catalogs: true
I20250115 14:52:07.886212 1167 ts_tablet_manager.cc:1918] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.886479 1167 log.cc:1198] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9d8c53fdf16243f0a614e48ec4ac94f4
I20250115 14:52:07.886750 1167 ts_tablet_manager.cc:1939] T 9d8c53fdf16243f0a614e48ec4ac94f4 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:07.887086 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5f4c7f8a9cd745ea935c1997f991a1f1" } new_table_name: "default.Table_59bae91b591d4af2879199b9fbf81bf5" modify_external_catalogs: true
I20250115 14:52:07.887408 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 762) took 1879 ms. Trace:
I20250115 14:52:07.887485 1167 rpcz_store.cc:276] 0115 14:52:06.008085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:06.008141 (+ 56us) service_pool.cc:225] Handling call
0115 14:52:07.709226 (+1701085us) ts_tablet_manager.cc:1145] Deleting tablet 9d8c53fdf16243f0a614e48ec4ac94f4
0115 14:52:07.709235 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.794222 (+ 84987us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.886136 (+ 91914us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:07.887387 (+ 1251us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1700926}
I20250115 14:52:07.888759 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9e22bc322f6e4b8bbcbdb9e756b33180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:07.889125 1164 tablet_bootstrap.cc:492] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.889232 1164 ts_tablet_manager.cc:1397] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
W20250115 14:52:07.889236 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e22bc322f6e4b8bbcbdb9e756b33180 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.890206 1164 raft_consensus.cc:357] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.890368 1164 raft_consensus.cc:383] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.890414 1164 raft_consensus.cc:738] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.890336 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ea36c03e003e440d99efe3c4c33838fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:07.890592 1164 consensus_queue.cc:260] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:07.890756 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea36c03e003e440d99efe3c4c33838fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.891465 1164 ts_tablet_manager.cc:1428] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:07.892037 2200 leader_election.cc:290] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.892760 2200 raft_consensus.cc:2743] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:07.892993 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2f8249110b37422b8edaa6c0dab8eb60" } modify_external_catalogs: true
I20250115 14:52:07.893157 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2f8249110b37422b8edaa6c0dab8eb60" } modify_external_catalogs: true
I20250115 14:52:07.894408 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a0070c4d1be947df84bf19c1efb88fa1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.894852 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0070c4d1be947df84bf19c1efb88fa1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.895947 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9c613e4019394b9790eb4d917dc249c5" } new_table_name: "default.Table_60dfab3c6fc946b3afe6a4e168627e2c" modify_external_catalogs: true
I20250115 14:52:07.897215 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dd66b1c3bf36403bb0d9300ca8f318a5" } new_table_name: "default.Table_4f9bd44ad3304deab5705c87466189da" modify_external_catalogs: true
I20250115 14:52:07.906620 1164 tablet_bootstrap.cc:492] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.912602 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9853dcdfee5f43de84798cfa0bef674e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.913023 1166 tablet_replica.cc:331] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.913208 1166 raft_consensus.cc:2238] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.913352 1166 raft_consensus.cc:2267] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.914252 1166 ts_tablet_manager.cc:1905] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.914619 2322 raft_consensus.cc:491] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.914754 2322 raft_consensus.cc:513] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.915377 2322 leader_election.cc:290] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.915839 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d590c63d6e4d058fac50dc50010317" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:07.920192 1166 ts_tablet_manager.cc:1918] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.920850 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 01291d41a2b345b4bfd00e6be0fb0129 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:07.921573 1166 log.cc:1198] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9853dcdfee5f43de84798cfa0bef674e
I20250115 14:52:07.921918 1166 ts_tablet_manager.cc:1939] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.922585 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9853dcdfee5f43de84798cfa0bef674e (table default.Table_4ec686b46a094818bd820c4f3854220b [id=5dc3e53ee73a4441960dc231d02d163f]) successfully deleted
I20250115 14:52:07.923080 1166 tablet_replica.cc:331] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:07.923378 1166 raft_consensus.cc:2238] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.923573 1166 raft_consensus.cc:2267] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.924401 1166 ts_tablet_manager.cc:1905] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.924930 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d611f6594f554cfe99c5aa5f2f7ea0e2" } modify_external_catalogs: true
I20250115 14:52:07.925151 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d611f6594f554cfe99c5aa5f2f7ea0e2" } modify_external_catalogs: true
I20250115 14:52:07.927735 1168 ts_tablet_manager.cc:1428] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.319s user 0.008s sys 0.002s
I20250115 14:52:07.929004 2345 raft_consensus.cc:491] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:07.929319 2345 raft_consensus.cc:513] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.930074 2345 leader_election.cc:290] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:07.553491 1958 rpcz_store.cc:276] 0115 14:52:03.420620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:03.420673 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:06.817058 (+3396385us) raft_consensus.cc:1466] Updating replica for 271 ops
0115 14:52:06.860942 (+ 43884us) raft_consensus.cc:1529] Early marking committed up to index 0
0115 14:52:06.860952 (+ 10us) raft_consensus.cc:1534] Triggering prepare for 271 ops
0115 14:52:06.861736 (+ 784us) write_op.cc:270] Start()
0115 14:52:06.861798 (+ 62us) write_op.cc:276] Timestamp: P: 1736952718154889 usec, L: 0
0115 14:52:07.030081 (+168283us) write_op.cc:270] Start()
0115 14:52:07.030118 (+ 37us) write_op.cc:276] Timestamp: P: 1736952718223534 usec, L: 0
0115 14:52:07.030339 (+ 221us) write_op.cc:270] Start()
0115 14:52:07.030373 (+ 34us) write_op.cc:276] Timestamp: P: 1736952718227576 usec, L: 0
0115 14:52:07.030525 (+ 152us) write_op.cc:270] Start()
0115 14:52:07.030543 (+ 18us) write_op.cc:276] Timestamp: P: 1736952718228153 usec, L: 0
0115 14:52:07.030664 (+ 121us) write_op.cc:270] Start()
0115 14:52:07.030680 (+ 16us) write_op.cc:276] Timestamp: P: 1736952718228639 usec, L: 0
0115 14:52:07.030798 (+ 118us) write_op.cc:270] Start()
0115 14:52:07.030816 (+ 18us) write_op.cc:276] Timestamp: P: 1736952718229586 usec, L: 0
0115 14:52:07.030933 (+ 117us) write_op.cc:270] Start()
0115 14:52:07.031002 (+ 69us) write_op.cc:276] Timestamp: P: 1736952718230541 usec, L: 0
0115 14:52:07.031183 (+ 181us) write_op.cc:270] Start()
0115 14:52:07.031206 (+ 23us) write_op.cc:276] Timestamp: P: 1736952718257399 usec, L: 0
0115 14:52:07.115088 (+ 83882us) write_op.cc:270] Start()
0115 14:52:07.115127 (+ 39us) write_op.cc:276] Timestamp: P: 1736952718258247 usec, L: 0
0115 14:52:07.115342 (+ 215us) write_op.cc:270] Start()
0115 14:52:07.115367 (+ 25us) write_op.cc:276] Timestamp: P: 1736952718270017 usec, L: 0
0115 14:52:07.115534 (+ 167us) write_op.cc:270] Start()
0115 14:52:07.115554 (+ 20us) write_op.cc:276] Timestamp: P: 1736952718272325 usec, L: 0
0115 14:52:07.115679 (+ 125us) write_op.cc:270] Start()
0115 14:52:07.115697 (+ 18us) write_op.cc:276] Timestamp: P: 1736952718632082 usec, L: 0
0115 14:52:07.115826 (+ 129us) write_op.cc:270] Start()
0115 14:52:07.115842 (+ 16us) write_op.cc:276] Timestamp: P: 1736952718646236 usec, L: 0
0115 14:52:07.115937 (+ 95us) write_op.cc:270] Start()
0115 14:52:07.115951 (+ 14us) write_op.cc:276] Timestamp: P: 1736952718654259 usec, L: 0
0115 14:52:07.116190 (+ 239us) write_op.cc:270] Start()
0115 14:52:07.116226 (+ 36us) write_op.cc:276] Timestamp: P: 1736952718794043 usec, L: 0
0115 14:52:07.116393 (+ 167us) write_op.cc:270] Start()
0115 14:52:07.116410 (+ 17us) write_op.cc:276] Timestamp: P: 1736952718797125 usec, L: 0
0115 14:52:07.116540 (+ 130us) write_op.cc:270] Start()
0115 14:52:07.116562 (+ 22us) write_op.cc:276] Timestamp: P: 1736952718802356 usec, L: 0
0115 14:52:07.116672 (+ 110us) write_op.cc:270] Start()
0115 14:52:07.116687 (+ 15us) write_op.cc:276] Timestamp: P: 1736952718802917 usec, L: 0
0115 14:52:07.116790 (+ 103us) write_op.cc:270] Start()
0115 14:52:07.116804 (+ 14us) write_op.cc:276] Timestamp: P: 1736952718815969 usec, L: 0
0115 14:52:07.116913 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.116928 (+ 15us) write_op.cc:276] Timestamp: P: 1736952718818462 usec, L: 0
0115 14:52:07.117055 (+ 127us) write_op.cc:270] Start()
0115 14:52:07.117071 (+ 16us) write_op.cc:276] Timestamp: P: 1736952718819552 usec, L: 0
0115 14:52:07.117179 (+ 108us) write_op.cc:270] Start()
0115 14:52:07.117196 (+ 17us) write_op.cc:276] Timestamp: P: 1736952718826173 usec, L: 0
0115 14:52:07.117309 (+ 113us) write_op.cc:270] Start()
0115 14:52:07.117325 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719061215 usec, L: 0
0115 14:52:07.117463 (+ 138us) write_op.cc:270] Start()
0115 14:52:07.117481 (+ 18us) write_op.cc:276] Timestamp: P: 1736952719064121 usec, L: 0
0115 14:52:07.117613 (+ 132us) write_op.cc:270] Start()
0115 14:52:07.117630 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719091272 usec, L: 0
0115 14:52:07.117738 (+ 108us) write_op.cc:270] Start()
0115 14:52:07.117753 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719096776 usec, L: 0
0115 14:52:07.117853 (+ 100us) write_op.cc:270] Start()
0115 14:52:07.117868 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719097664 usec, L: 0
0115 14:52:07.117977 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.117996 (+ 19us) write_op.cc:276] Timestamp: P: 1736952719100610 usec, L: 0
0115 14:52:07.118127 (+ 131us) write_op.cc:270] Start()
0115 14:52:07.118142 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719101175 usec, L: 0
0115 14:52:07.118239 (+ 97us) write_op.cc:270] Start()
0115 14:52:07.118254 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719104051 usec, L: 0
0115 14:52:07.118357 (+ 103us) write_op.cc:270] Start()
0115 14:52:07.118371 (+ 14us) write_op.cc:276] Timestamp: P: 1736952719157640 usec, L: 0
0115 14:52:07.118465 (+ 94us) write_op.cc:270] Start()
0115 14:52:07.118479 (+ 14us) write_op.cc:276] Timestamp: P: 1736952719158947 usec, L: 0
0115 14:52:07.118588 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.118603 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719190197 usec, L: 0
0115 14:52:07.118702 (+ 99us) write_op.cc:270] Start()
0115 14:52:07.118717 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719190770 usec, L: 0
0115 14:52:07.118859 (+ 142us) write_op.cc:270] Start()
0115 14:52:07.118876 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719191297 usec, L: 0
0115 14:52:07.118993 (+ 117us) write_op.cc:270] Start()
0115 14:52:07.119011 (+ 18us) write_op.cc:276] Timestamp: P: 1736952719191821 usec, L: 0
0115 14:52:07.119126 (+ 115us) write_op.cc:270] Start()
0115 14:52:07.119151 (+ 25us) write_op.cc:276] Timestamp: P: 1736952719285201 usec, L: 0
0115 14:52:07.119270 (+ 119us) write_op.cc:270] Start()
0115 14:52:07.119286 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719286238 usec, L: 0
0115 14:52:07.119405 (+ 119us) write_op.cc:270] Start()
0115 14:52:07.119425 (+ 20us) write_op.cc:276] Timestamp: P: 1736952719287895 usec, L: 0
0115 14:52:07.119525 (+ 100us) write_op.cc:270] Start()
0115 14:52:07.119539 (+ 14us) write_op.cc:276] Timestamp: P: 1736952719288478 usec, L: 0
0115 14:52:07.119636 (+ 97us) write_op.cc:270] Start()
0115 14:52:07.119650 (+ 14us) write_op.cc:276] Timestamp: P: 1736952719288988 usec, L: 0
0115 14:52:07.119751 (+ 101us) write_op.cc:270] Start()
0115 14:52:07.119768 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719289560 usec, L: 0
0115 14:52:07.119894 (+ 126us) write_op.cc:270] Start()
0115 14:52:07.119910 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719320128 usec, L: 0
0115 14:52:07.120020 (+ 110us) write_op.cc:270] Start()
0115 14:52:07.120034 (+ 14us) write_op.cc:276] Timestamp: P: 1736952719320805 usec, L: 0
0115 14:52:07.120138 (+ 104us) write_op.cc:270] Start()
0115 14:52:07.120154 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719321445 usec, L: 0
0115 14:52:07.120253 (+ 99us) write_op.cc:270] Start()
0115 14:52:07.120268 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719322103 usec, L: 0
0115 14:52:07.220043 (+ 99775us) write_op.cc:270] Start()
0115 14:52:07.220078 (+ 35us) write_op.cc:276] Timestamp: P: 1736952719322606 usec, L: 0
0115 14:52:07.220230 (+ 152us) write_op.cc:270] Start()
0115 14:52:07.220248 (+ 18us) write_op.cc:276] Timestamp: P: 1736952719323144 usec, L: 0
0115 14:52:07.220402 (+ 154us) write_op.cc:270] Start()
0115 14:52:07.220421 (+ 19us) write_op.cc:276] Timestamp: P: 1736952719331605 usec, L: 0
0115 14:52:07.220558 (+ 137us) write_op.cc:270] Start()
0115 14:52:07.220575 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719334431 usec, L: 0
0115 14:52:07.220927 (+ 352us) write_op.cc:270] Start()
0115 14:52:07.220951 (+ 24us) write_op.cc:276] Timestamp: P: 1736952719335014 usec, L: 0
0115 14:52:07.221094 (+ 143us) write_op.cc:270] Start()
0115 14:52:07.221112 (+ 18us) write_op.cc:276] Timestamp: P: 1736952719377547 usec, L: 0
0115 14:52:07.221224 (+ 112us) write_op.cc:270] Start()
0115 14:52:07.221241 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719379465 usec, L: 0
0115 14:52:07.221358 (+ 117us) write_op.cc:270] Start()
0115 14:52:07.221374 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719426179 usec, L: 0
0115 14:52:07.221470 (+ 96us) write_op.cc:270] Start()
0115 14:52:07.221484 (+ 14us) write_op.cc:276] Timestamp: P: 1736952719431628 usec, L: 0
0115 14:52:07.221580 (+ 96us) write_op.cc:270] Start()
0115 14:52:07.221595 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719432169 usec, L: 0
0115 14:52:07.221691 (+ 96us) write_op.cc:270] Start()
0115 14:52:07.221707 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719435174 usec, L: 0
0115 14:52:07.263594 (+ 41887us) write_op.cc:270] Start()
0115 14:52:07.263627 (+ 33us) write_op.cc:276] Timestamp: P: 1736952719462681 usec, L: 0
0115 14:52:07.263805 (+ 178us) write_op.cc:270] Start()
0115 14:52:07.263828 (+ 23us) write_op.cc:276] Timestamp: P: 1736952719504007 usec, L: 0
0115 14:52:07.264375 (+ 547us) write_op.cc:270] Start()
0115 14:52:07.264430 (+ 55us) write_op.cc:276] Timestamp: P: 1736952719522995 usec, L: 0
0115 14:52:07.264982 (+ 552us) write_op.cc:270] Start()
0115 14:52:07.265006 (+ 24us) write_op.cc:276] Timestamp: P: 1736952719529982 usec, L: 0
0115 14:52:07.265156 (+ 150us) write_op.cc:270] Start()
0115 14:52:07.265176 (+ 20us) write_op.cc:276] Timestamp: P: 1736952719533087 usec, L: 0
0115 14:52:07.265302 (+ 126us) write_op.cc:270] Start()
0115 14:52:07.265331 (+ 29us) write_op.cc:276] Timestamp: P: 1736952719539202 usec, L: 0
0115 14:52:07.265472 (+ 141us) write_op.cc:270] Start()
0115 14:52:07.265492 (+ 20us) write_op.cc:276] Timestamp: P: 1736952719542319 usec, L: 0
0115 14:52:07.265622 (+ 130us) write_op.cc:270] Start()
0115 14:52:07.265639 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719580163 usec, L: 0
0115 14:52:07.265748 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.265764 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719583556 usec, L: 0
0115 14:52:07.265873 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.265888 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719585394 usec, L: 0
0115 14:52:07.266002 (+ 114us) write_op.cc:270] Start()
0115 14:52:07.266019 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719595521 usec, L: 0
0115 14:52:07.266126 (+ 107us) write_op.cc:270] Start()
0115 14:52:07.266141 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719640878 usec, L: 0
0115 14:52:07.266246 (+ 105us) write_op.cc:270] Start()
0115 14:52:07.266262 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719725479 usec, L: 0
0115 14:52:07.266390 (+ 128us) write_op.cc:270] Start()
0115 14:52:07.266407 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719726217 usec, L: 0
0115 14:52:07.266524 (+ 117us) write_op.cc:270] Start()
0115 14:52:07.266540 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719726802 usec, L: 0
0115 14:52:07.266635 (+ 95us) write_op.cc:270] Start()
0115 14:52:07.266650 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719730123 usec, L: 0
0115 14:52:07.266765 (+ 115us) write_op.cc:270] Start()
0115 14:52:07.266782 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719759850 usec, L: 0
0115 14:52:07.266920 (+ 138us) write_op.cc:270] Start()
0115 14:52:07.266939 (+ 19us) write_op.cc:276] Timestamp: P: 1736952719760649 usec, L: 0
0115 14:52:07.267080 (+ 141us) write_op.cc:270] Start()
0115 14:52:07.267097 (+ 17us) write_op.cc:276] Timestamp: P: 1736952719826397 usec, L: 0
0115 14:52:07.267214 (+ 117us) write_op.cc:270] Start()
0115 14:52:07.267230 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719827079 usec, L: 0
0115 14:52:07.267339 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.267355 (+ 16us) write_op.cc:276] Timestamp: P: 1736952719870938 usec, L: 0
0115 14:52:07.267449 (+ 94us) write_op.cc:270] Start()
0115 14:52:07.267481 (+ 32us) write_op.cc:276] Timestamp: P: 1736952719871540 usec, L: 0
0115 14:52:07.267604 (+ 123us) write_op.cc:270] Start()
0115 14:52:07.267623 (+ 19us) write_op.cc:276] Timestamp: P: 1736952719871980 usec, L: 0
0115 14:52:07.267763 (+ 140us) write_op.cc:270] Start()
0115 14:52:07.267781 (+ 18us) write_op.cc:276] Timestamp: P: 1736952719872437 usec, L: 0
0115 14:52:07.267900 (+ 119us) write_op.cc:270] Start()
0115 14:52:07.267915 (+ 15us) write_op.cc:276] Timestamp: P: 1736952719872861 usec, L: 0
0115 14:52:07.268036 (+ 121us) write_op.cc:270] Start()
0115 14:52:07.268055 (+ 19us) write_op.cc:276] Timestamp: P: 1736952719873306 usec, L: 0
0115 14:52:07.298046 (+ 29991us) write_op.cc:270] Start()
0115 14:52:07.298088 (+ 42us) write_op.cc:276] Timestamp: P: 1736952719873696 usec, L: 0
0115 14:52:07.298311 (+ 223us) write_op.cc:270] Start()
0115 14:52:07.298336 (+ 25us) write_op.cc:276] Timestamp: P: 1736952719921339 usec, L: 0
0115 14:52:07.298483 (+ 147us) write_op.cc:270] Start()
0115 14:52:07.298506 (+ 23us) write_op.cc:276] Timestamp: P: 1736952719921924 usec, L: 0
0115 14:52:07.298657 (+ 151us) write_op.cc:270] Start()
0115 14:52:07.298679 (+ 22us) write_op.cc:276] Timestamp: P: 1736952719979236 usec, L: 0
0115 14:52:07.298822 (+ 143us) write_op.cc:270] Start()
0115 14:52:07.298843 (+ 21us) write_op.cc:276] Timestamp: P: 1736952719982596 usec, L: 0
0115 14:52:07.298980 (+ 137us) write_op.cc:270] Start()
0115 14:52:07.299000 (+ 20us) write_op.cc:276] Timestamp: P: 1736952719985621 usec, L: 0
0115 14:52:07.299130 (+ 130us) write_op.cc:270] Start()
0115 14:52:07.299147 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720003035 usec, L: 0
0115 14:52:07.299267 (+ 120us) write_op.cc:270] Start()
0115 14:52:07.299286 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720012684 usec, L: 0
0115 14:52:07.309763 (+ 10477us) write_op.cc:270] Start()
0115 14:52:07.309789 (+ 26us) write_op.cc:276] Timestamp: P: 1736952720069259 usec, L: 0
0115 14:52:07.309953 (+ 164us) write_op.cc:270] Start()
0115 14:52:07.309984 (+ 31us) write_op.cc:276] Timestamp: P: 1736952720071702 usec, L: 0
0115 14:52:07.310133 (+ 149us) write_op.cc:270] Start()
0115 14:52:07.310153 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720123105 usec, L: 0
0115 14:52:07.310262 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.310279 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720134455 usec, L: 0
0115 14:52:07.310407 (+ 128us) write_op.cc:270] Start()
0115 14:52:07.310423 (+ 16us) write_op.cc:276] Timestamp: P: 1736952720140721 usec, L: 0
0115 14:52:07.310521 (+ 98us) write_op.cc:270] Start()
0115 14:52:07.310537 (+ 16us) write_op.cc:276] Timestamp: P: 1736952720142921 usec, L: 0
0115 14:52:07.310649 (+ 112us) write_op.cc:270] Start()
0115 14:52:07.310665 (+ 16us) write_op.cc:276] Timestamp: P: 1736952720143483 usec, L: 0
0115 14:52:07.310799 (+ 134us) write_op.cc:270] Start()
0115 14:52:07.310819 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720144078 usec, L: 0
0115 14:52:07.313578 (+ 2759us) write_op.cc:270] Start()
0115 14:52:07.313605 (+ 27us) write_op.cc:276] Timestamp: P: 1736952720144693 usec, L: 0
0115 14:52:07.319863 (+ 6258us) write_op.cc:270] Start()
0115 14:52:07.319890 (+ 27us) write_op.cc:276] Timestamp: P: 1736952720145178 usec, L: 0
0115 14:52:07.320061 (+ 171us) write_op.cc:270] Start()
0115 14:52:07.320084 (+ 23us) write_op.cc:276] Timestamp: P: 1736952720145672 usec, L: 0
0115 14:52:07.320224 (+ 140us) write_op.cc:270] Start()
0115 14:52:07.320245 (+ 21us) write_op.cc:276] Timestamp: P: 1736952720146169 usec, L: 0
0115 14:52:07.320410 (+ 165us) write_op.cc:270] Start()
0115 14:52:07.320429 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720181932 usec, L: 0
0115 14:52:07.320563 (+ 134us) write_op.cc:270] Start()
0115 14:52:07.320583 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720189305 usec, L: 0
0115 14:52:07.320714 (+ 131us) write_op.cc:270] Start()
0115 14:52:07.320734 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720196763 usec, L: 0
0115 14:52:07.320863 (+ 129us) write_op.cc:270] Start()
0115 14:52:07.320880 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720206442 usec, L: 0
0115 14:52:07.321010 (+ 130us) write_op.cc:270] Start()
0115 14:52:07.321027 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720211457 usec, L: 0
0115 14:52:07.321157 (+ 130us) write_op.cc:270] Start()
0115 14:52:07.321176 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720228379 usec, L: 0
0115 14:52:07.321308 (+ 132us) write_op.cc:270] Start()
0115 14:52:07.321328 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720238113 usec, L: 0
0115 14:52:07.321449 (+ 121us) write_op.cc:270] Start()
0115 14:52:07.321466 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720240759 usec, L: 0
0115 14:52:07.321575 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.321593 (+ 18us) write_op.cc:276] Timestamp: P: 1736952720243778 usec, L: 0
0115 14:52:07.321712 (+ 119us) write_op.cc:270] Start()
0115 14:52:07.321730 (+ 18us) write_op.cc:276] Timestamp: P: 1736952720249621 usec, L: 0
0115 14:52:07.321839 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.321854 (+ 15us) write_op.cc:276] Timestamp: P: 1736952720315869 usec, L: 0
0115 14:52:07.321954 (+ 100us) write_op.cc:270] Start()
0115 14:52:07.321982 (+ 28us) write_op.cc:276] Timestamp: P: 1736952720333157 usec, L: 0
0115 14:52:07.322110 (+ 128us) write_op.cc:270] Start()
0115 14:52:07.322129 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720334785 usec, L: 0
0115 14:52:07.322247 (+ 118us) write_op.cc:270] Start()
0115 14:52:07.322264 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720421883 usec, L: 0
0115 14:52:07.322378 (+ 114us) write_op.cc:270] Start()
0115 14:52:07.322400 (+ 22us) write_op.cc:276] Timestamp: P: 1736952720422457 usec, L: 0
0115 14:52:07.322515 (+ 115us) write_op.cc:270] Start()
0115 14:52:07.322532 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720423035 usec, L: 0
0115 14:52:07.322650 (+ 118us) write_op.cc:270] Start()
0115 14:52:07.322668 (+ 18us) write_op.cc:276] Timestamp: P: 1736952720423555 usec, L: 0
0115 14:52:07.322785 (+ 117us) write_op.cc:270] Start()
0115 14:52:07.322803 (+ 18us) write_op.cc:276] Timestamp: P: 1736952720424051 usec, L: 0
0115 14:52:07.322944 (+ 141us) write_op.cc:270] Start()
0115 14:52:07.322964 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720424656 usec, L: 0
0115 14:52:07.323113 (+ 149us) write_op.cc:270] Start()
0115 14:52:07.323134 (+ 21us) write_op.cc:276] Timestamp: P: 1736952720425166 usec, L: 0
0115 14:52:07.323279 (+ 145us) write_op.cc:270] Start()
0115 14:52:07.323298 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720425639 usec, L: 0
0115 14:52:07.323423 (+ 125us) write_op.cc:270] Start()
0115 14:52:07.323442 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720427029 usec, L: 0
0115 14:52:07.323592 (+ 150us) write_op.cc:270] Start()
0115 14:52:07.323612 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720427939 usec, L: 0
0115 14:52:07.323731 (+ 119us) write_op.cc:270] Start()
0115 14:52:07.323750 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720500544 usec, L: 0
0115 14:52:07.323913 (+ 163us) write_op.cc:270] Start()
0115 14:52:07.323932 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720501106 usec, L: 0
0115 14:52:07.324088 (+ 156us) write_op.cc:270] Start()
0115 14:52:07.324108 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720504323 usec, L: 0
0115 14:52:07.324247 (+ 139us) write_op.cc:270] Start()
0115 14:52:07.324277 (+ 30us) write_op.cc:276] Timestamp: P: 1736952720517770 usec, L: 0
0115 14:52:07.324595 (+ 318us) write_op.cc:270] Start()
0115 14:52:07.324635 (+ 40us) write_op.cc:276] Timestamp: P: 1736952720520607 usec, L: 0
0115 14:52:07.324757 (+ 122us) write_op.cc:270] Start()
0115 14:52:07.324778 (+ 21us) write_op.cc:276] Timestamp: P: 1736952720521151 usec, L: 0
0115 14:52:07.324899 (+ 121us) write_op.cc:270] Start()
0115 14:52:07.324916 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720521722 usec, L: 0
0115 14:52:07.325059 (+ 143us) write_op.cc:270] Start()
0115 14:52:07.325079 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720523132 usec, L: 0
0115 14:52:07.325215 (+ 136us) write_op.cc:270] Start()
0115 14:52:07.325235 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720543413 usec, L: 0
0115 14:52:07.325348 (+ 113us) write_op.cc:270] Start()
0115 14:52:07.325365 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720598787 usec, L: 0
0115 14:52:07.325463 (+ 98us) write_op.cc:270] Start()
0115 14:52:07.325478 (+ 15us) write_op.cc:276] Timestamp: P: 1736952720624494 usec, L: 0
0115 14:52:07.325581 (+ 103us) write_op.cc:270] Start()
0115 14:52:07.325596 (+ 15us) write_op.cc:276] Timestamp: P: 1736952720693537 usec, L: 0
0115 14:52:07.325697 (+ 101us) write_op.cc:270] Start()
0115 14:52:07.325713 (+ 16us) write_op.cc:276] Timestamp: P: 1736952720724160 usec, L: 0
0115 14:52:07.325833 (+ 120us) write_op.cc:270] Start()
0115 14:52:07.325849 (+ 16us) write_op.cc:276] Timestamp: P: 1736952720725276 usec, L: 0
0115 14:52:07.325955 (+ 106us) write_op.cc:270] Start()
0115 14:52:07.325980 (+ 25us) write_op.cc:276] Timestamp: P: 1736952720732023 usec, L: 0
0115 14:52:07.326085 (+ 105us) write_op.cc:270] Start()
0115 14:52:07.326106 (+ 21us) write_op.cc:276] Timestamp: P: 1736952720732666 usec, L: 0
0115 14:52:07.326202 (+ 96us) write_op.cc:270] Start()
0115 14:52:07.326217 (+ 15us) write_op.cc:276] Timestamp: P: 1736952720733216 usec, L: 0
0115 14:52:07.326352 (+ 135us) write_op.cc:270] Start()
0115 14:52:07.326371 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720735745 usec, L: 0
0115 14:52:07.326506 (+ 135us) write_op.cc:270] Start()
0115 14:52:07.326525 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720738418 usec, L: 0
0115 14:52:07.326639 (+ 114us) write_op.cc:270] Start()
0115 14:52:07.326654 (+ 15us) write_op.cc:276] Timestamp: P: 1736952720864202 usec, L: 0
0115 14:52:07.326756 (+ 102us) write_op.cc:270] Start()
0115 14:52:07.326771 (+ 15us) write_op.cc:276] Timestamp: P: 1736952720865287 usec, L: 0
0115 14:52:07.326902 (+ 131us) write_op.cc:270] Start()
0115 14:52:07.326918 (+ 16us) write_op.cc:276] Timestamp: P: 1736952720866299 usec, L: 0
0115 14:52:07.327034 (+ 116us) write_op.cc:270] Start()
0115 14:52:07.327050 (+ 16us) write_op.cc:276] Timestamp: P: 1736952720867130 usec, L: 0
0115 14:52:07.327152 (+ 102us) write_op.cc:270] Start()
0115 14:52:07.327167 (+ 15us) write_op.cc:276] Timestamp: P: 1736952720868001 usec, L: 0
0115 14:52:07.327281 (+ 114us) write_op.cc:270] Start()
0115 14:52:07.327300 (+ 19us) write_op.cc:276] Timestamp: P: 1736952720870509 usec, L: 0
0115 14:52:07.327443 (+ 143us) write_op.cc:270] Start()
0115 14:52:07.327463 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720882073 usec, L: 0
0115 14:52:07.327589 (+ 126us) write_op.cc:270] Start()
0115 14:52:07.327609 (+ 20us) write_op.cc:276] Timestamp: P: 1736952720923401 usec, L: 0
0115 14:52:07.327735 (+ 126us) write_op.cc:270] Start()
0115 14:52:07.327753 (+ 18us) write_op.cc:276] Timestamp: P: 1736952720954292 usec, L: 0
0115 14:52:07.327865 (+ 112us) write_op.cc:270] Start()
0115 14:52:07.327882 (+ 17us) write_op.cc:276] Timestamp: P: 1736952720955312 usec, L: 0
0115 14:52:07.328000 (+ 118us) write_op.cc:270] Start()
0115 14:52:07.328015 (+ 15us) write_op.cc:276] Timestamp: P: 1736952720955957 usec, L: 0
0115 14:52:07.328136 (+ 121us) write_op.cc:270] Start()
0115 14:52:07.328153 (+ 17us) write_op.cc:276] Timestamp: P: 1736952721053079 usec, L: 0
0115 14:52:07.328256 (+ 103us) write_op.cc:270] Start()
0115 14:52:07.328271 (+ 15us) write_op.cc:276] Timestamp: P: 1736952721069500 usec, L: 0
0115 14:52:07.328401 (+ 130us) write_op.cc:270] Start()
0115 14:52:07.328425 (+ 24us) write_op.cc:276] Timestamp: P: 1736952721092900 usec, L: 0
0115 14:52:07.328547 (+ 122us) write_op.cc:270] Start()
0115 14:52:07.328564 (+ 17us) write_op.cc:276] Timestamp: P: 1736952721113285 usec, L: 0
0115 14:52:07.328689 (+ 125us) write_op.cc:270] Start()
0115 14:52:07.328725 (+ 36us) write_op.cc:276] Timestamp: P: 1736952721136255 usec, L: 0
0115 14:52:07.328869 (+ 144us) write_op.cc:270] Start()
0115 14:52:07.328890 (+ 21us) write_op.cc:276] Timestamp: P: 1736952721137681 usec, L: 0
0115 14:52:07.366303 (+ 37413us) write_op.cc:270] Start()
0115 14:52:07.366350 (+ 47us) write_op.cc:276] Timestamp: P: 1736952721152853 usec, L: 0
0115 14:52:07.366552 (+ 202us) write_op.cc:270] Start()
0115 14:52:07.366573 (+ 21us) write_op.cc:276] Timestamp: P: 1736952721154559 usec, L: 0
0115 14:52:07.366708 (+ 135us) write_op.cc:270] Start()
0115 14:52:07.366726 (+ 18us) write_op.cc:276] Timestamp: P: 1736952721323496 usec, L: 0
0115 14:52:07.366849 (+ 123us) write_op.cc:270] Start()
0115 14:52:07.366866 (+ 17us) write_op.cc:276] Timestamp: P: 1736952721325012 usec, L: 0
0115 14:52:07.367000 (+ 134us) write_op.cc:270] Start()
0115 14:52:07.367058 (+ 58us) write_op.cc:276] Timestamp: P: 1736952721325602 usec, L: 0
0115 14:52:07.367226 (+ 168us) write_op.cc:270] Start()
0115 14:52:07.367243 (+ 17us) write_op.cc:276] Timestamp: P: 1736952721355332 usec, L: 0
0115 14:52:07.368621 (+ 1378us) write_op.cc:270] Start()
0115 14:52:07.368650 (+ 29us) write_op.cc:276] Timestamp: P: 1736952721356397 usec, L: 0
0115 14:52:07.368837 (+ 187us) write_op.cc:270] Start()
0115 14:52:07.368857 (+ 20us) write_op.cc:276] Timestamp: P: 1736952721357380 usec, L: 0
0115 14:52:07.369017 (+ 160us) write_op.cc:270] Start()
0115 14:52:07.369036 (+ 19us) write_op.cc:276] Timestamp: P: 1736952721358158 usec, L: 0
0115 14:52:07.369156 (+ 120us) write_op.cc:270] Start()
0115 14:52:07.369172 (+ 16us) write_op.cc:276] Timestamp: P: 1736952721358943 usec, L: 0
0115 14:52:07.369300 (+ 128us) write_op.cc:270] Start()
0115 14:52:07.369320 (+ 20us) write_op.cc:276] Timestamp: P: 1736952721359671 usec, L: 0
0115 14:52:07.369440 (+ 120us) write_op.cc:270] Start()
0115 14:52:07.369457 (+ 17us) write_op.cc:276] Timestamp: P: 1736952721379077 usec, L: 0
0115 14:52:07.369568 (+ 111us) write_op.cc:270] Start()
0115 14:52:07.369587 (+ 19us) write_op.cc:276] Timestamp: P: 1736952721439208 usec, L: 0
0115 14:52:07.369754 (+ 167us) write_op.cc:270] Start()
0115 14:52:07.369779 (+ 25us) write_op.cc:276] Timestamp: P: 1736952721511999 usec, L: 0
0115 14:52:07.369944 (+ 165us) write_op.cc:270] Start()
0115 14:52:07.369977 (+ 33us) write_op.cc:276] Timestamp: P: 1736952721514360 usec, L: 0
0115 14:52:07.370112 (+ 135us) write_op.cc:270] Start()
0115 14:52:07.370131 (+ 19us) write_op.cc:276] Timestamp: P: 1736952721528192 usec, L: 0
0115 14:52:07.370257 (+ 126us) write_op.cc:270] Start()
0115 14:52:07.370275 (+ 18us) write_op.cc:276] Timestamp: P: 1736952721534491 usec, L: 0
0115 14:52:07.370397 (+ 122us) write_op.cc:270] Start()
0115 14:52:07.370413 (+ 16us) write_op.cc:276] Timestamp: P: 1736952721536702 usec, L: 0
0115 14:52:07.370539 (+ 126us) write_op.cc:270] Start()
0115 14:52:07.370557 (+ 18us) write_op.cc:276] Timestamp: P: 1736952721538245 usec, L: 0
0115 14:52:07.370671 (+ 114us) write_op.cc:270] Start()
0115 14:52:07.370686 (+ 15us) write_op.cc:276] Timestamp: P: 1736952721540052 usec, L: 0
0115 14:52:07.370815 (+ 129us) write_op.cc:270] Start()
0115 14:52:07.370834 (+ 19us) write_op.cc:276] Timestamp: P: 1736952721547153 usec, L: 0
0115 14:52:07.370959 (+ 125us) write_op.cc:270] Start()
0115 14:52:07.370988 (+ 29us) write_op.cc:276] Timestamp: P: 1736952721548771 usec, L: 0
0115 14:52:07.371131 (+ 143us) write_op.cc:270] Start()
0115 14:52:07.371153 (+ 22us) write_op.cc:276] Timestamp: P: 1736952721552644 usec, L: 0
0115 14:52:07.371277 (+ 124us) write_op.cc:270] Start()
0115 14:52:07.371295 (+ 18us) write_op.cc:276] Timestamp: P: 1736952721630621 usec, L: 0
0115 14:52:07.371423 (+ 128us) write_op.cc:270] Start()
0115 14:52:07.371443 (+ 20us) write_op.cc:276] Timestamp: P: 1736952721631283 usec, L: 0
0115 14:52:07.371579 (+ 136us) write_op.cc:270] Start()
0115 14:52:07.371596 (+ 17us) write_op.cc:276] Timestamp: P: 1736952721632580 usec, L: 0
0115 14:52:07.371712 (+ 116us) write_op.cc:270] Start()
0115 14:52:07.371731 (+ 19us) write_op.cc:276] Timestamp: P: 1736952721672130 usec, L: 0
0115 14:52:07.371840 (+ 109us) write_op.cc:270] Start()
0115 14:52:07.371857 (+ 17us) write_op.cc:276] Timestamp: P: 1736952721675066 usec, L: 0
0115 14:52:07.372006 (+ 149us) write_op.cc:270] Start()
0115 14:52:07.372026 (+ 20us) write_op.cc:276] Timestamp: P: 1736952721675927 usec, L: 0
0115 14:52:07.372167 (+ 141us) write_op.cc:270] Start()
0115 14:52:07.372186 (+ 19us) write_op.cc:276] Timestamp: P: 1736952721737299 usec, L: 0
0115 14:52:07.385104 (+ 12918us) write_op.cc:270] Start()
0115 14:52:07.385137 (+ 33us) write_op.cc:276] Timestamp: P: 1736952721752919 usec, L: 0
0115 14:52:07.385326 (+ 189us) write_op.cc:270] Start()
0115 14:52:07.385350 (+ 24us) write_op.cc:276] Timestamp: P: 1736952721774309 usec, L: 0
0115 14:52:07.385494 (+ 144us) write_op.cc:270] Start()
0115 14:52:07.385516 (+ 22us) write_op.cc:276] Timestamp: P: 1736952721776619 usec, L: 0
0115 14:52:07.385665 (+ 149us) write_op.cc:270] Start()
0115 14:52:07.385685 (+ 20us) write_op.cc:276] Timestamp: P: 1736952721778897 usec, L: 0
0115 14:52:07.385809 (+ 124us) write_op.cc:270] Start()
0115 14:52:07.385829 (+ 20us) write_op.cc:276] Timestamp: P: 1736952
I20250115 14:52:07.932552 1164 tablet_bootstrap.cc:654] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:07.932899 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cee072daff2d4943b68261806666e4a0 consensus state has no leader
I20250115 14:52:07.932909 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8ab6a34caa486ab929af2cac3ce957" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:07.933099 1049 raft_consensus.cc:3054] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:07.933336 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ab10a36d2dd14df0b5b7dde01478a485" } new_table_name: "default.Table_8ece818885104a7c9e659b32feff419f" modify_external_catalogs: true
I20250115 14:52:07.934604 1166 ts_tablet_manager.cc:1918] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:07.934791 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 894532f9a4d542a09eb51c2fcdd2907a consensus state has no leader
I20250115 14:52:07.935168 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f2006bfc8694432482d41dd92edbf093"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:07.935454 1049 raft_consensus.cc:2463] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
W20250115 14:52:07.935622 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2006bfc8694432482d41dd92edbf093 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.936993 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3962d1f342824bf990b968aeba430bff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:07.937458 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3962d1f342824bf990b968aeba430bff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:07.941291 1166 log.cc:1198] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/01291d41a2b345b4bfd00e6be0fb0129
I20250115 14:52:07.941823 1166 ts_tablet_manager.cc:1939] T 01291d41a2b345b4bfd00e6be0fb0129 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:07.942016 1167 tablet_replica.cc:331] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:07.942028 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d590c63d6e4d058fac50dc50010317" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:07.942426 1167 raft_consensus.cc:2238] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.942567 1167 raft_consensus.cc:2267] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.943074 1167 ts_tablet_manager.cc:1905] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.943077 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 01291d41a2b345b4bfd00e6be0fb0129 (table default.Table_21fa1a9206d64c8bb19a405f37518a65 [id=e31104604e8648c08fd706e44150cda1]) successfully deleted
I20250115 14:52:07.941134 1164 tablet_bootstrap.cc:492] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.943325 1164 ts_tablet_manager.cc:1397] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.001s
I20250115 14:52:07.944025 1164 raft_consensus.cc:357] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.944206 1164 raft_consensus.cc:383] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.944262 1164 raft_consensus.cc:738] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.944605 1162 tablet_bootstrap.cc:492] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:07.944731 1162 ts_tablet_manager.cc:1397] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.002s
I20250115 14:52:07.946223 1164 consensus_queue.cc:260] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:07.948356 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9853dcdfee5f43de84798cfa0bef674e" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:07.948529 908 raft_consensus.cc:2463] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:07.952144 740 tablet_service.cc:1467] Processing CreateTablet for tablet b6fb388c17044f5eabce9996ebbf4ff6 (DEFAULT_TABLE table=default.Table_3caaaaf71eae487480e4b55631f7e099 [id=3e9726d47fae47dda6aa17b27df72b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.952527 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fb388c17044f5eabce9996ebbf4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.956595 1168 tablet_bootstrap.cc:492] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:07.957441 771 raft_consensus.cc:3054] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:07.958207 1168 tablet_bootstrap.cc:654] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.960625 2273 consensus_queue.cc:1035] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:07.961233 1164 ts_tablet_manager.cc:1428] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250115 14:52:07.963332 710 leader_election.cc:304] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:07.963738 1164 tablet_bootstrap.cc:492] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:07.964833 2329 tablet.cc:1721] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:07.964022 710 leader_election.cc:304] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
W20250115 14:52:07.967769 710 raft_consensus.cc:2681] T 9853dcdfee5f43de84798cfa0bef674e P d2d7ecece2ab49e1be7079216517d13b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250115 14:52:07.969429 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b75d759f834d558c94cc981e338ac0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:07.969681 1049 raft_consensus.cc:2463] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:07.969869 1164 tablet_bootstrap.cc:654] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:07.971112 1162 raft_consensus.cc:357] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:07.972572 1162 raft_consensus.cc:383] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:07.972631 1162 raft_consensus.cc:738] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:07.973776 740 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0eeede8f714b92b1e3a9a558289c3e (DEFAULT_TABLE table=default.Table_13267a7581594c869ec943c3a7f1058e [id=1d02fc79a4424e83b550af0e64cf7739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.974411 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f0eeede8f714b92b1e3a9a558289c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.977087 1751 tablet_replica.cc:331] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:07.977279 1751 raft_consensus.cc:2238] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.977416 1751 raft_consensus.cc:2267] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.977888 1751 ts_tablet_manager.cc:1905] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.978108 1024 tablet_service.cc:1467] Processing CreateTablet for tablet f2ba4086a6bf4bfa83e740b31dfb53f8 (DEFAULT_TABLE table=default.Table_c2aefc9f5c604090bb4bd4f2326991ba [id=62ff1172217947669bf02f96134877c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.978423 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ba4086a6bf4bfa83e740b31dfb53f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.980508 748 tablet_service.cc:1467] Processing CreateTablet for tablet f2ba4086a6bf4bfa83e740b31dfb53f8 (DEFAULT_TABLE table=default.Table_c2aefc9f5c604090bb4bd4f2326991ba [id=62ff1172217947669bf02f96134877c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.980847 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ba4086a6bf4bfa83e740b31dfb53f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:07.980985 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ba33eb3f5a40450db5cf115fbe8a17e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba33eb3f5a40450db5cf115fbe8a17e7 already in progress: creating tablet
W20250115 14:52:07.981204 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 580742eb5dac4dba9995809890e0e315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 580742eb5dac4dba9995809890e0e315 already in progress: creating tablet
W20250115 14:52:07.981420 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b803d8e103a5423f85c75c9006809b17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b803d8e103a5423f85c75c9006809b17 already in progress: creating tablet
W20250115 14:52:07.984503 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 140977us
I20250115 14:52:07.984566 1751 ts_tablet_manager.cc:1918] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:07.984536 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 8c545ef40b4743c89b9e9165982d4b2f (DEFAULT_TABLE table=default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.984650 1751 log.cc:1198] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e5a059e2f0654a4aafe9e8082b8fde7a
I20250115 14:52:07.984874 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c545ef40b4743c89b9e9165982d4b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.984901 1751 ts_tablet_manager.cc:1939] T e5a059e2f0654a4aafe9e8082b8fde7a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:07.986589 1751 tablet_replica.cc:331] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:07.986752 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e5a059e2f0654a4aafe9e8082b8fde7a (table default.Table_dbb5dd7ec474409b9743a74e69cc1b44 [id=54db87fdb55743b8a485808e00abdbee]) successfully deleted
W20250115 14:52:07.987120 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ded663ac1197454bbde3c4c2d526002d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ded663ac1197454bbde3c4c2d526002d already in progress: creating tablet
I20250115 14:52:07.987862 1751 raft_consensus.cc:2238] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:07.988842 1751 raft_consensus.cc:2267] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:07.990393 1751 ts_tablet_manager.cc:1905] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:07.992686 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0eeede8f714b92b1e3a9a558289c3e (DEFAULT_TABLE table=default.Table_13267a7581594c869ec943c3a7f1058e [id=1d02fc79a4424e83b550af0e64cf7739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.996104 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f0eeede8f714b92b1e3a9a558289c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.998035 1164 tablet_bootstrap.cc:492] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:07.998189 1164 ts_tablet_manager.cc:1397] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250115 14:52:07.998888 1168 tablet_bootstrap.cc:492] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:07.999176 1168 ts_tablet_manager.cc:1397] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.001s
I20250115 14:52:08.000174 847 leader_election.cc:304] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:08.001691 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea8ab6a34caa486ab929af2cac3ce957" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:08.001505 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.002427 1166 tablet_replica.cc:331] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:08.002322 1168 raft_consensus.cc:357] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.002491 1168 raft_consensus.cc:383] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.002578 1168 raft_consensus.cc:738] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.002722 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b75d759f834d558c94cc981e338ac0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.002768 1168 consensus_queue.cc:260] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.003139 1166 raft_consensus.cc:2238] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.003202 773 raft_consensus.cc:3054] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.003334 1166 raft_consensus.cc:2267] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.004029 1166 ts_tablet_manager.cc:1905] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.004423 879 tablet_service.cc:1467] Processing CreateTablet for tablet 3bef17ce7b43457bb11dba0bb1fd70dd (DEFAULT_TABLE table=default.Table_3950c52dccfb443bbfd1bed7a0007bd7 [id=6a7b69f7dd9a4b799eff42705854708c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.004773 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bef17ce7b43457bb11dba0bb1fd70dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.007407 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2f7a5cbd9a78485a816d7fecdc23106a
I20250115 14:52:08.008543 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7a5cbd9a78485a816d7fecdc23106a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.010274 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9339e2f34b994dd7a9f6385ee32f1b05" } modify_external_catalogs: true
I20250115 14:52:08.010372 2329 tablet.cc:1721] T 4b68c168048649c5b91600eb4302b0ce P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250115 14:52:08.010740 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29adc03e38624dfc910e07de24466d1b consensus state has no leader
I20250115 14:52:08.011472 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 005a4cd0e9f14e76b01afa49d946ee32
I20250115 14:52:08.011921 449 master_service.cc:946] ReplaceTablet: received request to replace tablet e910c387e6a24caf95e14b57c8b83e6f from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:08.013513 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 005a4cd0e9f14e76b01afa49d946ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.013648 1166 ts_tablet_manager.cc:1918] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.013738 1166 log.cc:1198] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/8e6cc08065494d9e85eeff5044b44ba6
I20250115 14:52:08.013995 1166 ts_tablet_manager.cc:1939] T 8e6cc08065494d9e85eeff5044b44ba6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:08.014333 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9339e2f34b994dd7a9f6385ee32f1b05" } modify_external_catalogs: true
I20250115 14:52:08.014770 893 tablet_service.cc:1467] Processing CreateTablet for tablet 8c545ef40b4743c89b9e9165982d4b2f (DEFAULT_TABLE table=default.Table_5b812a62d552484794327ad5be4f5286 [id=63c4e413417941a9ae951ece417e74ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.014876 891 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd405050a2548e1b9ea7d7bf34a35af (DEFAULT_TABLE table=default.Table_29a289769ec64b96a7352fa0412f999b [id=292730d3a3c5477bae35f28e717417a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.015101 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c545ef40b4743c89b9e9165982d4b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.014516 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 8e6cc08065494d9e85eeff5044b44ba6 (table default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]) successfully deleted
I20250115 14:52:08.015337 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fd405050a2548e1b9ea7d7bf34a35af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.017088 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9d8c53fdf16243f0a614e48ec4ac94f4 (table default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]) successfully deleted
I20250115 14:52:08.018150 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4b68c168048649c5b91600eb4302b0ce (table default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1]). Expected version 3 got 0
I20250115 14:52:08.018381 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]). Expected version 1 got 0
I20250115 14:52:08.018594 886 tablet_service.cc:1467] Processing CreateTablet for tablet b6fb388c17044f5eabce9996ebbf4ff6 (DEFAULT_TABLE table=default.Table_3caaaaf71eae487480e4b55631f7e099 [id=3e9726d47fae47dda6aa17b27df72b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.018893 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fb388c17044f5eabce9996ebbf4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:08.019932 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fab76e0427f14da48bf101db57f0457c consensus state has no leader
I20250115 14:52:08.020179 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 005a4cd0e9f14e76b01afa49d946ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.023734 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7a5cbd9a78485a816d7fecdc23106a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.000419 1164 raft_consensus.cc:357] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.024215 1164 raft_consensus.cc:383] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.024276 1164 raft_consensus.cc:738] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.024510 1164 consensus_queue.cc:260] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.025033 769 raft_consensus.cc:3054] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.026155 2322 raft_consensus.cc:491] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.026268 2322 raft_consensus.cc:513] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.026870 2322 leader_election.cc:290] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.027017 879 tablet_service.cc:1467] Processing CreateTablet for tablet 83edbad0640c4bf39f4497522c4135e4 (DEFAULT_TABLE table=default.Table_86158e4855654f77bfa28259f0d0e38b [id=c3903ce80530492590797926806ec07f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.027324 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83edbad0640c4bf39f4497522c4135e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.027498 769 raft_consensus.cc:2463] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:08.028196 2241 raft_consensus.cc:491] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.028333 2241 raft_consensus.cc:513] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.028951 2241 leader_election.cc:290] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.029109 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_76b7b4a0c17041068bfb2be07cf21267" } modify_external_catalogs: true
I20250115 14:52:08.029284 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_76b7b4a0c17041068bfb2be07cf21267" } modify_external_catalogs: true
I20250115 14:52:08.030542 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e842e1c2b44754b3b9a2e026486a84" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.031132 984 leader_election.cc:343] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.031296 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a38d44069676412aa914ae1ff4ca0861"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.031450 847 leader_election.cc:304] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:08.031435 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338c0f6f39fd4335bbcda733b28b8bf8" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:52:08.031854 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a38d44069676412aa914ae1ff4ca0861 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.032481 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e842e1c2b44754b3b9a2e026486a84" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.032759 911 raft_consensus.cc:2463] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:08.033151 983 leader_election.cc:304] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.033354 2326 tablet.cc:1721] T 4b68c168048649c5b91600eb4302b0ce P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:08.033581 2241 raft_consensus.cc:2798] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:08.033648 2241 raft_consensus.cc:491] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:08.033702 2241 raft_consensus.cc:3054] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.034389 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_944d3194b69f4c6a9c0899a204051442" } modify_external_catalogs: true
I20250115 14:52:08.034533 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_944d3194b69f4c6a9c0899a204051442" } modify_external_catalogs: true
I20250115 14:52:08.035892 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b29e828cea18463e9e2b28ab06b9f1aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.036135 2241 raft_consensus.cc:513] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:08.036422 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b29e828cea18463e9e2b28ab06b9f1aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.036628 2340 raft_consensus.cc:2798] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:08.036757 2340 raft_consensus.cc:695] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:08.036991 2340 consensus_queue.cc:237] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.037539 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e842e1c2b44754b3b9a2e026486a84" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:08.037549 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2c0b01c3b5394ebbac4a5e930276ed6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.037940 2138 raft_consensus.cc:2798] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250115 14:52:08.037962 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c0b01c3b5394ebbac4a5e930276ed6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.038007 2138 raft_consensus.cc:491] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250115 14:52:08.038022 984 leader_election.cc:343] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.038071 2138 raft_consensus.cc:3054] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.038558 2241 leader_election.cc:290] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.038782 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e842e1c2b44754b3b9a2e026486a84" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:08.038969 908 raft_consensus.cc:3054] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:07.993367 1162 consensus_queue.cc:260] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.039670 2273 raft_consensus.cc:491] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.039764 2273 raft_consensus.cc:513] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.040386 2273 leader_election.cc:290] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.040771 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b599c6b8174959b8b66e90e9242057" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.040820 1843 raft_consensus.cc:491] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.040925 1843 raft_consensus.cc:513] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:08.041160 983 leader_election.cc:343] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.040776 2138 raft_consensus.cc:513] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.042014 2138 leader_election.cc:290] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.042292 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b599c6b8174959b8b66e90e9242057" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.042579 1751 ts_tablet_manager.cc:1918] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.042681 1751 log.cc:1198] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/90c1602aef514dd29d5d2cffb043dbb6
W20250115 14:52:08.042749 984 leader_election.cc:343] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.042833 984 leader_election.cc:304] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.042928 1751 ts_tablet_manager.cc:1939] T 90c1602aef514dd29d5d2cffb043dbb6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.043175 2273 raft_consensus.cc:2743] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.044171 1162 ts_tablet_manager.cc:1428] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.099s user 0.001s sys 0.001s
I20250115 14:52:08.045073 1843 leader_election.cc:290] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.045161 908 raft_consensus.cc:2463] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:08.045570 983 leader_election.cc:304] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.045944 2273 raft_consensus.cc:2798] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:08.046039 2273 raft_consensus.cc:695] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:08.046264 2273 consensus_queue.cc:237] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.048112 886 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0eeede8f714b92b1e3a9a558289c3e (DEFAULT_TABLE table=default.Table_13267a7581594c869ec943c3a7f1058e [id=1d02fc79a4424e83b550af0e64cf7739]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.048367 893 tablet_service.cc:1467] Processing CreateTablet for tablet f2ba4086a6bf4bfa83e740b31dfb53f8 (DEFAULT_TABLE table=default.Table_c2aefc9f5c604090bb4bd4f2326991ba [id=62ff1172217947669bf02f96134877c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.048528 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f0eeede8f714b92b1e3a9a558289c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.048484 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 90c1602aef514dd29d5d2cffb043dbb6 (table default.Table_950b48d1b1344479a7229041f044bbb6 [id=65eabb2be7d1479f889da99e50da551f]) successfully deleted
I20250115 14:52:08.048676 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ba4086a6bf4bfa83e740b31dfb53f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:07.993960 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b6fb388c17044f5eabce9996ebbf4ff6 (DEFAULT_TABLE table=default.Table_3caaaaf71eae487480e4b55631f7e099 [id=3e9726d47fae47dda6aa17b27df72b5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:07.992864 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a000ab73dd46ac8e134784a156040a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.052265 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fb388c17044f5eabce9996ebbf4ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.053331 1751 tablet_replica.cc:331] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.053627 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f7a5cbd9a78485a816d7fecdc23106a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.054111 1751 raft_consensus.cc:2238] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.054515 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b75d759f834d558c94cc981e338ac0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:08.054926 1047 raft_consensus.cc:3054] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.054803 1162 tablet_bootstrap.cc:492] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.055823 2324 raft_consensus.cc:2743] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.056780 1162 tablet_bootstrap.cc:654] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.057714 1047 raft_consensus.cc:2463] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:08.058230 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "338c0f6f39fd4335bbcda733b28b8bf8" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.058830 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 005a4cd0e9f14e76b01afa49d946ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.060460 847 leader_election.cc:304] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:08.060933 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0d97fa57484b9d8cb879fedda61612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:08.062757 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d443d39716d04b45b3151be80ed4320b consensus state has no leader
I20250115 14:52:08.062914 2341 raft_consensus.cc:2798] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:08.063019 2341 raft_consensus.cc:695] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:08.063226 2341 consensus_queue.cc:237] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:08.065295 847 leader_election.cc:343] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 4abde784f9c948cb96e6cc93c30817bd
I20250115 14:52:08.065310 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abde784f9c948cb96e6cc93c30817bd" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.065580 710 leader_election.cc:304] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
W20250115 14:52:08.066679 848 leader_election.cc:343] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.067350 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e6cc08065494d9e85eeff5044b44ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.067660 2324 raft_consensus.cc:2743] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:08.069394 848 leader_election.cc:343] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.069490 848 leader_election.cc:304] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.054936 1751 raft_consensus.cc:2267] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.070236 1751 ts_tablet_manager.cc:1905] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.072800 2138 raft_consensus.cc:2743] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.073130 1164 ts_tablet_manager.cc:1428] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.075s user 0.002s sys 0.001s
I20250115 14:52:08.073830 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3b75d759f834d558c94cc981e338ac0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:08.074641 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e910c387e6a24caf95e14b57c8b83e6f
I20250115 14:52:08.074363 2325 tablet.cc:1721] T 4b68c168048649c5b91600eb4302b0ce P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:08.075135 449 catalog_manager.cc:6596] ReplaceTablet: tablet e910c387e6a24caf95e14b57c8b83e6f deleted and replaced by tablet bb859e2a553f4ab3b20452b48e933f0e
I20250115 14:52:08.076484 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b54807a4e9644b86a89670f57ae154a3
I20250115 14:52:08.072562 2241 raft_consensus.cc:491] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.075457 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet e910c387e6a24caf95e14b57c8b83e6f with delete_type TABLET_DATA_DELETED (replaced by tablet bb859e2a553f4ab3b20452b48e933f0e) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.077215 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet b54807a4e9644b86a89670f57ae154a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.079250 2241 raft_consensus.cc:513] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.079866 2241 leader_election.cc:290] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.082180 748 tablet_service.cc:1514] Processing DeleteTablet for tablet b54807a4e9644b86a89670f57ae154a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.082401 740 tablet_service.cc:1514] Processing DeleteTablet for tablet e910c387e6a24caf95e14b57c8b83e6f with delete_type TABLET_DATA_DELETED (replaced by tablet bb859e2a553f4ab3b20452b48e933f0e) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.082840 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.083323 984 leader_election.cc:343] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.083621 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:08.084316 983 leader_election.cc:343] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.084420 983 leader_election.cc:304] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.084831 2241 raft_consensus.cc:2743] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.074630 1164 tablet_bootstrap.cc:492] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:08.086357 882 tablet_service.cc:1514] Processing DeleteTablet for tablet e910c387e6a24caf95e14b57c8b83e6f with delete_type TABLET_DATA_DELETED (replaced by tablet bb859e2a553f4ab3b20452b48e933f0e) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.086931 893 tablet_service.cc:1514] Processing DeleteTablet for tablet b54807a4e9644b86a89670f57ae154a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.087334 771 raft_consensus.cc:3054] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:08.091552 848 consensus_peers.cc:487] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80c75a77d7154b5cb05a5d9ac8a20787. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:08.091956 1164 tablet_bootstrap.cc:654] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.091887 1770 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:52:08.092144 1770 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: This master's current role is: LEADER
I20250115 14:52:08.095074 1167 ts_tablet_manager.cc:1918] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.095793 1162 tablet_bootstrap.cc:492] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.095935 1162 ts_tablet_manager.cc:1397] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20250115 14:52:08.096748 1162 raft_consensus.cc:357] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.096935 1162 raft_consensus.cc:383] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.096992 1162 raft_consensus.cc:738] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.097222 1162 consensus_queue.cc:260] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.097440 1164 tablet_bootstrap.cc:492] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:08.097770 1162 ts_tablet_manager.cc:1428] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:08.097913 1932 ts_manager.cc:194] Registered new tserver with Master: 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.098054 1162 tablet_bootstrap.cc:492] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.099820 1162 tablet_bootstrap.cc:654] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.097573 1164 ts_tablet_manager.cc:1397] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250115 14:52:08.104161 1164 raft_consensus.cc:357] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.104389 1164 raft_consensus.cc:383] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.104445 1164 raft_consensus.cc:738] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.105500 1167 log.cc:1198] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d71a1ecd900942e5ae788c9a28c70a7a
I20250115 14:52:08.105881 1167 ts_tablet_manager.cc:1939] T d71a1ecd900942e5ae788c9a28c70a7a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:08.106141 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 763) took 2097 ms. Trace:
I20250115 14:52:08.106222 1167 rpcz_store.cc:276] 0115 14:52:06.008248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:06.012378 (+ 4130us) service_pool.cc:225] Handling call
0115 14:52:07.887770 (+1875392us) ts_tablet_manager.cc:1145] Deleting tablet d71a1ecd900942e5ae788c9a28c70a7a
0115 14:52:07.887780 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:07.945935 (+ 58155us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.094973 (+149038us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.106127 (+ 11154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871547}
I20250115 14:52:08.106555 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 795) took 1364 ms. Trace:
I20250115 14:52:08.106627 1167 rpcz_store.cc:276] 0115 14:52:06.741884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:06.746697 (+ 4813us) service_pool.cc:225] Handling call
0115 14:52:08.106457 (+1359760us) ts_tablet_manager.cc:1145] Deleting tablet abccc613e1754839a4c7a8f5c448a852
0115 14:52:08.106465 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.106542 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359606}
I20250115 14:52:08.106925 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 796) took 1364 ms. Trace:
I20250115 14:52:08.106988 1167 rpcz_store.cc:276] 0115 14:52:06.742943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:06.746876 (+ 3933us) service_pool.cc:225] Handling call
0115 14:52:08.106837 (+1359961us) ts_tablet_manager.cc:1145] Deleting tablet c14836b97c3247b39ff9072c12e642e0
0115 14:52:08.106844 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.106909 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359841}
I20250115 14:52:08.107234 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 797) took 1363 ms. Trace:
I20250115 14:52:08.107290 1167 rpcz_store.cc:276] 0115 14:52:06.743663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:06.747169 (+ 3506us) service_pool.cc:225] Handling call
0115 14:52:08.107159 (+1359990us) ts_tablet_manager.cc:1145] Deleting tablet f60e68b411364ada9b89879012d862ea
0115 14:52:08.107166 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.107224 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359857}
I20250115 14:52:08.107482 1167 tablet_replica.cc:331] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:08.107672 1167 raft_consensus.cc:2238] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.107717 2325 tablet.cc:1721] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:08.107829 1167 raft_consensus.cc:2267] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.108525 1167 ts_tablet_manager.cc:1905] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.108502 1164 consensus_queue.cc:260] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.103399 1168 ts_tablet_manager.cc:1428] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.104s user 0.013s sys 0.001s
I20250115 14:52:08.109009 1168 tablet_bootstrap.cc:492] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:08.112512 848 leader_election.cc:343] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.116788 1164 ts_tablet_manager.cc:1428] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250115 14:52:08.117081 1164 tablet_bootstrap.cc:492] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:08.118499 1164 tablet_bootstrap.cc:654] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.120584 1162 tablet_bootstrap.cc:492] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.120705 1162 ts_tablet_manager.cc:1397] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.002s
I20250115 14:52:08.121392 1162 raft_consensus.cc:357] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.121575 1162 raft_consensus.cc:383] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.121631 1162 raft_consensus.cc:738] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.121850 1162 consensus_queue.cc:260] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.122422 1162 ts_tablet_manager.cc:1428] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:08.122666 1162 tablet_bootstrap.cc:492] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.124377 1162 tablet_bootstrap.cc:654] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.126480 771 raft_consensus.cc:3054] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.134912 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d71a1ecd900942e5ae788c9a28c70a7a (table default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]) successfully deleted
W20250115 14:52:08.136055 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 92aeca346a7746468dbbff053d933472 (table default.Table_b3e4daa02d5e4d8ca7f0710e6c5f8260 [id=60ff7f730d394990a0263e322d28d7ff], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92aeca346a7746468dbbff053d933472 consensus state has no leader
W20250115 14:52:08.136708 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet abccc613e1754839a4c7a8f5c448a852 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abccc613e1754839a4c7a8f5c448a852 already in progress: creating tablet
W20250115 14:52:08.137248 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c14836b97c3247b39ff9072c12e642e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c14836b97c3247b39ff9072c12e642e0 already in progress: creating tablet
W20250115 14:52:08.137706 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f60e68b411364ada9b89879012d862ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f60e68b411364ada9b89879012d862ea already in progress: creating tablet
I20250115 14:52:08.139197 455 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_ebfbdc698c8c448ca2e60757b1fd574c [id=471b029f1dba4dd087deb339bba02013]). Expected version 2 got 0
I20250115 14:52:08.141695 448 catalog_manager.cc:5526] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:08.142480 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ea8ab6a34caa486ab929af2cac3ce957 (table default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d]). Expected version 1 got 0
W20250115 14:52:08.143815 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2f7a5cbd9a78485a816d7fecdc23106a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f7a5cbd9a78485a816d7fecdc23106a already in progress: creating tablet
W20250115 14:52:08.144762 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 005a4cd0e9f14e76b01afa49d946ee32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 005a4cd0e9f14e76b01afa49d946ee32 already in progress: creating tablet
W20250115 14:52:08.144974 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 8e6cc08065494d9e85eeff5044b44ba6 because the tablet was not found. No further retry: Not found: Tablet not found: 8e6cc08065494d9e85eeff5044b44ba6
W20250115 14:52:08.145145 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b54807a4e9644b86a89670f57ae154a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b54807a4e9644b86a89670f57ae154a3 already in progress: creating tablet
W20250115 14:52:08.145285 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e910c387e6a24caf95e14b57c8b83e6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e910c387e6a24caf95e14b57c8b83e6f already in progress: creating tablet
I20250115 14:52:08.145512 1168 tablet_bootstrap.cc:654] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.145545 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2ffb08cfa3fa4eea9582ee724bc15f9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.146032 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ffb08cfa3fa4eea9582ee724bc15f9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.146943 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fd4d720185214c7cbec302eaf573e265"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.147383 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd4d720185214c7cbec302eaf573e265 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.150372 450 master_service.cc:946] ReplaceTablet: received request to replace tablet e5dfc00ecec54f92a75fa77c84fe8a6d from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:08.151535 1167 ts_tablet_manager.cc:1918] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.152251 1164 tablet_bootstrap.cc:492] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:08.152402 1164 ts_tablet_manager.cc:1397] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20250115 14:52:08.153108 1164 raft_consensus.cc:357] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.153295 1164 raft_consensus.cc:383] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.153345 1164 raft_consensus.cc:738] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.153545 1164 consensus_queue.cc:260] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.154135 1162 tablet_bootstrap.cc:492] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.154271 1162 ts_tablet_manager.cc:1397] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.003s
I20250115 14:52:08.155622 1162 raft_consensus.cc:357] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.155809 1162 raft_consensus.cc:383] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.155864 1162 raft_consensus.cc:738] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.156086 1162 consensus_queue.cc:260] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.125835 1751 ts_tablet_manager.cc:1918] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.156767 1751 log.cc:1198] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/282d988bc41443c381bb0ba69b618f5d
I20250115 14:52:08.157085 1751 ts_tablet_manager.cc:1939] T 282d988bc41443c381bb0ba69b618f5d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.158035 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 282d988bc41443c381bb0ba69b618f5d (table default.Table_4ab29505c6634887aa99118862eccf5d [id=83320921ba86467b9ad31fde698cb2d1]) successfully deleted
W20250115 14:52:08.158659 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 79736602ea2646b39708ca9c6834e847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79736602ea2646b39708ca9c6834e847 already in progress: creating tablet
I20250115 14:52:08.159850 2324 raft_consensus.cc:491] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.159963 2324 raft_consensus.cc:513] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.160524 2324 leader_election.cc:290] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.160570 1162 ts_tablet_manager.cc:1428] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20250115 14:52:08.161254 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee8b63cc21145d8993b058d8c6d23c7" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:08.161413 1047 raft_consensus.cc:2463] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:08.163173 710 leader_election.cc:304] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:08.163415 2324 raft_consensus.cc:2798] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:08.163478 2324 raft_consensus.cc:491] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:08.163532 2324 raft_consensus.cc:3054] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.164677 1162 tablet_bootstrap.cc:492] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.165900 2324 raft_consensus.cc:513] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.166469 2324 leader_election.cc:290] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.167214 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee8b63cc21145d8993b058d8c6d23c7" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:08.167356 1047 raft_consensus.cc:3054] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:08.168407 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 40eae66b35974524992b2ef943273291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40eae66b35974524992b2ef943273291 already in progress: creating tablet
W20250115 14:52:08.169165 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7ff71ded3b074ff0979e2090131f4eaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ff71ded3b074ff0979e2090131f4eaa already in progress: creating tablet
I20250115 14:52:08.169759 1047 raft_consensus.cc:2463] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
W20250115 14:52:08.170166 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 338706b2d7514f94afd3579da458b4ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 338706b2d7514f94afd3579da458b4ec already in progress: creating tablet
W20250115 14:52:08.170666 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b579be5c683e4fd8a5d5484fdb89250c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b579be5c683e4fd8a5d5484fdb89250c already in progress: creating tablet
W20250115 14:52:08.171186 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 064f890830304758a6840a1502e59080 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 064f890830304758a6840a1502e59080 already in progress: creating tablet
I20250115 14:52:08.171343 710 leader_election.cc:304] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:08.171428 1751 tablet_replica.cc:331] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.171597 1751 raft_consensus.cc:2238] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.171761 1751 raft_consensus.cc:2267] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.171804 2324 raft_consensus.cc:2798] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:08.171993 2324 raft_consensus.cc:695] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:08.172214 2324 consensus_queue.cc:237] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.172446 2273 raft_consensus.cc:491] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.172541 2273 raft_consensus.cc:513] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.172643 1751 ts_tablet_manager.cc:1905] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.173039 2273 leader_election.cc:290] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.174224 2273 raft_consensus.cc:491] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.174326 2273 raft_consensus.cc:513] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.174856 2273 leader_election.cc:290] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.175144 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377d711f052545ddbe499afe7ba3de28" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.175597 984 leader_election.cc:343] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.176641 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5937d6ad0494734a402fdc594eb9415" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.177170 984 leader_election.cc:343] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.181102 1164 ts_tablet_manager.cc:1428] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.029s user 0.001s sys 0.000s
I20250115 14:52:08.181386 1164 tablet_bootstrap.cc:492] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:08.182755 1164 tablet_bootstrap.cc:654] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.192584 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d360add9d39a48caa7b90f7d786f7bc1
I20250115 14:52:08.193854 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d360add9d39a48caa7b90f7d786f7bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.194557 1162 tablet_bootstrap.cc:654] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.195961 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 210f161cf6b94a2abbad651c40bb8cec
I20250115 14:52:08.197057 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 210f161cf6b94a2abbad651c40bb8cec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.197607 740 tablet_service.cc:1514] Processing DeleteTablet for tablet d360add9d39a48caa7b90f7d786f7bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.170477 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee8b63cc21145d8993b058d8c6d23c7" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.170863 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ee8b63cc21145d8993b058d8c6d23c7" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:08.173750 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5937d6ad0494734a402fdc594eb9415" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:08.200214 983 leader_election.cc:343] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.200341 983 leader_election.cc:304] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.175774 1168 tablet_bootstrap.cc:492] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:08.200683 2273 raft_consensus.cc:2743] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.200767 1168 ts_tablet_manager.cc:1397] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.001s
I20250115 14:52:08.201397 1168 raft_consensus.cc:357] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.201578 1168 raft_consensus.cc:383] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.201318 451 catalog_manager.cc:5526] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:08.201630 1168 raft_consensus.cc:738] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.201850 1168 consensus_queue.cc:260] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.202435 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_06f9926d94994f899b4cb94f5a89f817" } new_table_name: "default.Table_93b115e2ed72434093f43964aadb2636" modify_external_catalogs: true
I20250115 14:52:08.203601 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fa4c201ba84d4cf5902269c4c6e4f50c" } new_table_name: "default.Table_e5bbdf72c53c402e8a5fa2955799247b" modify_external_catalogs: true
I20250115 14:52:08.204708 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e5e8956be33c4ba7ab1e001e9719a2ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.205153 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5e8956be33c4ba7ab1e001e9719a2ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.205363 1168 ts_tablet_manager.cc:1428] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:08.205614 1168 tablet_bootstrap.cc:492] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.193683 879 tablet_service.cc:1514] Processing DeleteTablet for tablet d360add9d39a48caa7b90f7d786f7bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.196918 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 210f161cf6b94a2abbad651c40bb8cec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.176096 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377d711f052545ddbe499afe7ba3de28" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:08.206672 983 leader_election.cc:343] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.206789 983 leader_election.cc:304] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.207141 2273 raft_consensus.cc:2743] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.208194 1168 tablet_bootstrap.cc:654] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.208467 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5a3e279adb9548baa2d7264f270977e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.209017 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a3e279adb9548baa2d7264f270977e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.210063 1167 log.cc:1198] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/104282a09c024bbaafeb5f28f91dadcb
I20250115 14:52:08.210350 1167 ts_tablet_manager.cc:1939] T 104282a09c024bbaafeb5f28f91dadcb P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:08.210661 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 798) took 1466 ms. Trace:
I20250115 14:52:08.210745 1167 rpcz_store.cc:276] 0115 14:52:06.744646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:06.758394 (+ 13748us) service_pool.cc:225] Handling call
0115 14:52:08.107450 (+1349056us) ts_tablet_manager.cc:1145] Deleting tablet 104282a09c024bbaafeb5f28f91dadcb
0115 14:52:08.107457 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.148802 (+ 41345us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.151457 (+ 2655us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.210647 (+ 59190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348812}
I20250115 14:52:08.211097 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 104282a09c024bbaafeb5f28f91dadcb (table default.Table_06a952903c7b4dcc8088b8c9ff5a2ac5 [id=09205f90c7fc476185a1d31a60ad5fa0]) successfully deleted
W20250115 14:52:08.211450 710 leader_election.cc:343] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:08.211807 710 leader_election.cc:343] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.212261 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 210f161cf6b94a2abbad651c40bb8cec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:08.213109 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d360add9d39a48caa7b90f7d786f7bc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d360add9d39a48caa7b90f7d786f7bc1 already in progress: creating tablet
W20250115 14:52:08.213716 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 210f161cf6b94a2abbad651c40bb8cec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 210f161cf6b94a2abbad651c40bb8cec already in progress: creating tablet
I20250115 14:52:08.219337 2325 tablet.cc:1721] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:08.219787 1162 tablet_bootstrap.cc:492] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.219916 1162 ts_tablet_manager.cc:1397] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.001s
I20250115 14:52:08.220803 1162 raft_consensus.cc:357] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.220988 1162 raft_consensus.cc:383] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.221040 1162 raft_consensus.cc:738] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.221254 1162 consensus_queue.cc:260] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.221710 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 813) took 1127 ms. Trace:
I20250115 14:52:08.221794 1167 rpcz_store.cc:276] 0115 14:52:07.094300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.094341 (+ 41us) service_pool.cc:225] Handling call
0115 14:52:08.221591 (+1127250us) ts_tablet_manager.cc:1145] Deleting tablet ba33eb3f5a40450db5cf115fbe8a17e7
0115 14:52:08.221602 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.221691 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1126962}
W20250115 14:52:08.222154 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ba33eb3f5a40450db5cf115fbe8a17e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba33eb3f5a40450db5cf115fbe8a17e7 already in progress: creating tablet
I20250115 14:52:08.222729 1167 tablet_replica.cc:331] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:08.222898 1167 raft_consensus.cc:2238] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.223021 1167 raft_consensus.cc:2267] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.223486 1167 ts_tablet_manager.cc:1905] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.223829 1162 ts_tablet_manager.cc:1428] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:52:08.224084 1162 tablet_bootstrap.cc:492] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.225673 1162 tablet_bootstrap.cc:654] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.232252 1168 tablet_bootstrap.cc:492] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:08.232400 1168 ts_tablet_manager.cc:1397] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250115 14:52:08.232988 1168 raft_consensus.cc:357] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.233155 1168 raft_consensus.cc:383] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.233194 1168 raft_consensus.cc:738] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.233516 1168 consensus_queue.cc:260] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.234059 1751 ts_tablet_manager.cc:1918] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.237905 1168 ts_tablet_manager.cc:1428] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250115 14:52:08.241816 1168 tablet_bootstrap.cc:492] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.241475 1751 log.cc:1198] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/efb8b001dd66456d9e49e9e8a6848ccf
I20250115 14:52:08.242291 1751 ts_tablet_manager.cc:1939] T efb8b001dd66456d9e49e9e8a6848ccf P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.243196 1168 tablet_bootstrap.cc:654] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.244052 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7db2f6193de84d23a01b711b7de04815" } modify_external_catalogs: true
I20250115 14:52:08.244268 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7db2f6193de84d23a01b711b7de04815" } modify_external_catalogs: true
I20250115 14:52:08.246806 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet efb8b001dd66456d9e49e9e8a6848ccf (table default.Table_e8f8acbb30db4f019ebeadda783b434a [id=877cb5d7df394006822f9a0e8a133dcc]) successfully deleted
W20250115 14:52:08.247561 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 78a2d39d0296468db48b3eec374244ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a2d39d0296468db48b3eec374244ca already in progress: creating tablet
W20250115 14:52:08.248268 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f5c1f0e0b6ee4448b9ba2f78d0117197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5c1f0e0b6ee4448b9ba2f78d0117197 already in progress: creating tablet
I20250115 14:52:08.248601 1751 tablet_replica.cc:331] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.248845 1751 raft_consensus.cc:2238] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.249024 1751 raft_consensus.cc:2267] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.250599 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4a8a9e000ade430aaa44bad5dd61df9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.251124 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a8a9e000ade430aaa44bad5dd61df9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.251948 1162 tablet_bootstrap.cc:492] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.252076 1162 ts_tablet_manager.cc:1397] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20250115 14:52:08.252761 1162 raft_consensus.cc:357] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.252928 1162 raft_consensus.cc:383] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.252980 1162 raft_consensus.cc:738] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.253159 1162 consensus_queue.cc:260] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.253604 1162 ts_tablet_manager.cc:1428] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250115 14:52:08.254318 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e5dfc00ecec54f92a75fa77c84fe8a6d
I20250115 14:52:08.254842 450 catalog_manager.cc:6596] ReplaceTablet: tablet e5dfc00ecec54f92a75fa77c84fe8a6d deleted and replaced by tablet 9107b85c08c646cab34f2128ea8db84f
I20250115 14:52:08.255149 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_519ee6346f5a476f875b3d4b5e249b0b" } new_table_name: "default.Table_8428a9aac32d42deb52a9b45f3520443" modify_external_catalogs: true
I20250115 14:52:08.255373 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e5dfc00ecec54f92a75fa77c84fe8a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 9107b85c08c646cab34f2128ea8db84f) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.255604 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet e5dfc00ecec54f92a75fa77c84fe8a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 9107b85c08c646cab34f2128ea8db84f) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.259223 740 tablet_service.cc:1514] Processing DeleteTablet for tablet e5dfc00ecec54f92a75fa77c84fe8a6d with delete_type TABLET_DATA_DELETED (replaced by tablet 9107b85c08c646cab34f2128ea8db84f) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:08.259999 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e5dfc00ecec54f92a75fa77c84fe8a6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5dfc00ecec54f92a75fa77c84fe8a6d already in progress: creating tablet
I20250115 14:52:08.260710 1751 ts_tablet_manager.cc:1905] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.266858 1167 ts_tablet_manager.cc:1918] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.266973 1167 log.cc:1198] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/7d07686c40324233a49e4c49aeed73f6
I20250115 14:52:08.267233 1167 ts_tablet_manager.cc:1939] T 7d07686c40324233a49e4c49aeed73f6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:08.267983 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 816) took 1173 ms. Trace:
I20250115 14:52:08.268074 1167 rpcz_store.cc:276] 0115 14:52:07.094572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.094611 (+ 39us) service_pool.cc:225] Handling call
0115 14:52:08.222675 (+1128064us) ts_tablet_manager.cc:1145] Deleting tablet 7d07686c40324233a49e4c49aeed73f6
0115 14:52:08.222685 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.241109 (+ 18424us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.266759 (+ 25650us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.267961 (+ 1202us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127927}
I20250115 14:52:08.268572 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 7d07686c40324233a49e4c49aeed73f6 (table default.Table_8f5c258dbdc04de79751c3a1b995e0fb [id=e71b0f487dc0428aaa03b4cc7cf95aa0]) successfully deleted
I20250115 14:52:08.268232 1168 tablet_bootstrap.cc:492] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:08.272480 1168 ts_tablet_manager.cc:1397] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20250115 14:52:08.273159 1168 raft_consensus.cc:357] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.273392 1168 raft_consensus.cc:383] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.276333 1168 raft_consensus.cc:738] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.276654 1168 consensus_queue.cc:260] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:08.277451 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 894532f9a4d542a09eb51c2fcdd2907a consensus state has no leader
I20250115 14:52:08.277812 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ea36c03e003e440d99efe3c4c33838fe" } modify_external_catalogs: true
I20250115 14:52:08.278010 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ea36c03e003e440d99efe3c4c33838fe" } modify_external_catalogs: true
W20250115 14:52:08.278566 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9aefb12a03784eb895a17fd8c571aa24 consensus state has no leader
I20250115 14:52:08.278923 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_340265c995a649808c582e4c4e6d0b22" } new_table_name: "default.Table_58f31090da1c45208ab0b73f1ceb0ba4" modify_external_catalogs: true
I20250115 14:52:08.279250 1164 tablet_bootstrap.cc:492] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:08.279379 1164 ts_tablet_manager.cc:1397] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.098s user 0.020s sys 0.007s
I20250115 14:52:08.280184 1162 tablet_bootstrap.cc:492] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.280336 452 master_service.cc:946] ReplaceTablet: received request to replace tablet a349cfaff6be4b85a0b2cf3eee238c26 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:08.280105 1164 raft_consensus.cc:357] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.280624 1164 raft_consensus.cc:383] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.280689 1164 raft_consensus.cc:738] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.280900 1164 consensus_queue.cc:260] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.282874 1168 ts_tablet_manager.cc:1428] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20250115 14:52:08.283257 1164 ts_tablet_manager.cc:1428] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:08.283514 1164 tablet_bootstrap.cc:492] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:08.285084 1164 tablet_bootstrap.cc:654] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.286453 2340 raft_consensus.cc:491] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.286571 2340 raft_consensus.cc:513] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.287171 2340 leader_election.cc:290] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.287482 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ce65604c294021af93632ed073f627" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.287957 847 leader_election.cc:343] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d9ce65604c294021af93632ed073f627
W20250115 14:52:08.288247 848 leader_election.cc:343] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.288388 848 leader_election.cc:304] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.288822 2340 raft_consensus.cc:2743] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.289214 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 817) took 1194 ms. Trace:
I20250115 14:52:08.289331 1167 rpcz_store.cc:276] 0115 14:52:07.094697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.094794 (+ 97us) service_pool.cc:225] Handling call
0115 14:52:08.289099 (+1194305us) ts_tablet_manager.cc:1145] Deleting tablet b803d8e103a5423f85c75c9006809b17
0115 14:52:08.289109 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.289198 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1194175}
W20250115 14:52:08.289844 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b803d8e103a5423f85c75c9006809b17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b803d8e103a5423f85c75c9006809b17 already in progress: creating tablet
I20250115 14:52:08.291945 1162 tablet_bootstrap.cc:654] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.292202 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 812) took 1198 ms. Trace:
I20250115 14:52:08.292371 1167 rpcz_store.cc:276] 0115 14:52:07.094158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.095177 (+ 1019us) service_pool.cc:225] Handling call
0115 14:52:08.292080 (+1196903us) ts_tablet_manager.cc:1145] Deleting tablet 580742eb5dac4dba9995809890e0e315
0115 14:52:08.292091 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.292185 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196775}
W20250115 14:52:08.293066 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 580742eb5dac4dba9995809890e0e315 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 580742eb5dac4dba9995809890e0e315 already in progress: creating tablet
I20250115 14:52:08.293428 1167 tablet_replica.cc:331] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:08.294461 1167 raft_consensus.cc:2238] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.294914 1167 raft_consensus.cc:2267] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.295879 1167 ts_tablet_manager.cc:1905] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.305207 2340 raft_consensus.cc:491] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.305349 2340 raft_consensus.cc:513] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.305984 2340 leader_election.cc:290] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.307054 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efb8b001dd66456d9e49e9e8a6848ccf" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.307843 848 leader_election.cc:343] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:08.309247 847 leader_election.cc:343] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: efb8b001dd66456d9e49e9e8a6848ccf
I20250115 14:52:08.309348 847 leader_election.cc:304] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.309906 2340 raft_consensus.cc:2743] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.310194 1751 ts_tablet_manager.cc:1918] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.310350 1751 log.cc:1198] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b5937d6ad0494734a402fdc594eb9415
I20250115 14:52:08.310705 1751 ts_tablet_manager.cc:1939] T b5937d6ad0494734a402fdc594eb9415 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.292272 1168 tablet_bootstrap.cc:492] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.310993 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e6612d24c40f44a780551d89e880543a" } new_table_name: "default.Table_241b0cb6766b476da1e9e3a989f69e8d" modify_external_catalogs: true
I20250115 14:52:08.315876 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 929830d8e8fd4b0c8a7b18c941706929
I20250115 14:52:08.316757 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_33efb40a15454adfb022b1d76c2aa409"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.317308 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33efb40a15454adfb022b1d76c2aa409 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.319053 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 929830d8e8fd4b0c8a7b18c941706929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:08.319766 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 929830d8e8fd4b0c8a7b18c941706929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 929830d8e8fd4b0c8a7b18c941706929 already in progress: creating tablet
I20250115 14:52:08.321672 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 929830d8e8fd4b0c8a7b18c941706929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.321259 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 929830d8e8fd4b0c8a7b18c941706929 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.324829 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b5937d6ad0494734a402fdc594eb9415 (table default.Table_c2aefc9f5c604090bb4bd4f2326991ba [id=62ff1172217947669bf02f96134877c3]) successfully deleted
I20250115 14:52:08.324990 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_50430a4659d544eba42d4e0e1cef37da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.325542 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50430a4659d544eba42d4e0e1cef37da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:08.326756 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d79fb3826ff483ea6224fe3c277521c consensus state has no leader
I20250115 14:52:08.327085 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2814144b33794c15b9386d76197e2e95" } modify_external_catalogs: true
I20250115 14:52:08.327260 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2814144b33794c15b9386d76197e2e95" } modify_external_catalogs: true
W20250115 14:52:08.327883 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b126874bae6e4c84bd97f5ff65e0882b consensus state has no leader
I20250115 14:52:08.328161 453 master_service.cc:946] ReplaceTablet: received request to replace tablet a48227ec5aa846c3ac63bd29ed38be25 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:08.328464 2340 raft_consensus.cc:491] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.328595 2340 raft_consensus.cc:513] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.329210 2340 leader_election.cc:290] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:08.330161 848 leader_election.cc:343] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Not found: Tablet not found: 8e6cc08065494d9e85eeff5044b44ba6
W20250115 14:52:08.330986 847 leader_election.cc:343] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 8e6cc08065494d9e85eeff5044b44ba6
I20250115 14:52:08.331076 847 leader_election.cc:304] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.331456 2340 raft_consensus.cc:2743] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.332511 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a349cfaff6be4b85a0b2cf3eee238c26
I20250115 14:52:08.332649 1162 tablet_bootstrap.cc:492] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.332948 452 catalog_manager.cc:6596] ReplaceTablet: tablet a349cfaff6be4b85a0b2cf3eee238c26 deleted and replaced by tablet c300667f775e40b5a65dce61f4863cae
I20250115 14:52:08.333997 886 tablet_service.cc:1514] Processing DeleteTablet for tablet a349cfaff6be4b85a0b2cf3eee238c26 with delete_type TABLET_DATA_DELETED (replaced by tablet c300667f775e40b5a65dce61f4863cae) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.334475 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet a349cfaff6be4b85a0b2cf3eee238c26 with delete_type TABLET_DATA_DELETED (replaced by tablet c300667f775e40b5a65dce61f4863cae) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.332780 1162 ts_tablet_manager.cc:1397] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.000s
I20250115 14:52:08.335238 740 tablet_service.cc:1514] Processing DeleteTablet for tablet a349cfaff6be4b85a0b2cf3eee238c26 with delete_type TABLET_DATA_DELETED (replaced by tablet c300667f775e40b5a65dce61f4863cae) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.335355 1162 raft_consensus.cc:357] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.335513 1162 raft_consensus.cc:383] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.335551 1162 raft_consensus.cc:738] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.335753 1162 consensus_queue.cc:260] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.336335 1162 ts_tablet_manager.cc:1428] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:08.336565 1162 tablet_bootstrap.cc:492] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:08.339090 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a349cfaff6be4b85a0b2cf3eee238c26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a349cfaff6be4b85a0b2cf3eee238c26 already in progress: creating tablet
I20250115 14:52:08.340045 1167 ts_tablet_manager.cc:1918] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.340158 1167 log.cc:1198] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/fc3d2ca7cd7c48a79ed1396fbbdc6866
W20250115 14:52:08.340653 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6e2d55a4e32448fbb98f704dbf1657b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2d55a4e32448fbb98f704dbf1657b9 already in progress: creating tablet
W20250115 14:52:08.341272 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 45f124134eab4babac145404c62e7d2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45f124134eab4babac145404c62e7d2a already in progress: creating tablet
I20250115 14:52:08.341531 1751 tablet_replica.cc:331] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.342054 1751 raft_consensus.cc:2238] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.342216 1751 raft_consensus.cc:2267] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.342880 1751 ts_tablet_manager.cc:1905] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.343683 1162 tablet_bootstrap.cc:654] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.346028 1168 tablet_bootstrap.cc:654] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.360435 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 1140016d11a74a2cadba494c0b319c54 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:08.361488 1167 ts_tablet_manager.cc:1939] T fc3d2ca7cd7c48a79ed1396fbbdc6866 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:08.362358 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 815) took 1267 ms. Trace:
I20250115 14:52:08.370790 1164 tablet_bootstrap.cc:492] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:08.371172 1164 ts_tablet_manager.cc:1397] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.088s user 0.035s sys 0.019s
W20250115 14:52:08.371814 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 47848.2us
I20250115 14:52:08.381129 1162 tablet_bootstrap.cc:492] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.381281 1162 ts_tablet_manager.cc:1397] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.002s
I20250115 14:52:08.382050 1162 raft_consensus.cc:357] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.382275 1162 raft_consensus.cc:383] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.382323 1162 raft_consensus.cc:738] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.382516 1162 consensus_queue.cc:260] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.383008 1162 ts_tablet_manager.cc:1428] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:08.383455 1162 tablet_bootstrap.cc:492] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.385097 1162 tablet_bootstrap.cc:654] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.385341 1168 tablet_bootstrap.cc:492] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:08.386161 1168 ts_tablet_manager.cc:1397] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.094s user 0.000s sys 0.004s
I20250115 14:52:08.386881 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet ea8ab6a34caa486ab929af2cac3ce957 (table default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d]). Expected version 1 got 0
I20250115 14:52:08.387650 1168 raft_consensus.cc:357] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.387796 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_eb5fca7507634f2d97b090af382eb6c9 [id=30b975fd37374a5a90d8cbda734489f1] alter complete (version 3)
I20250115 14:52:08.388254 1168 raft_consensus.cc:383] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.388346 1168 raft_consensus.cc:738] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.388536 1168 consensus_queue.cc:260] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.389060 1168 ts_tablet_manager.cc:1428] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250115 14:52:08.389324 1168 tablet_bootstrap.cc:492] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.389236 1164 raft_consensus.cc:357] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.389643 1164 raft_consensus.cc:383] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.389700 1164 raft_consensus.cc:738] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.389901 1164 consensus_queue.cc:260] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.390604 1168 tablet_bootstrap.cc:654] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.391132 1164 ts_tablet_manager.cc:1428] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250115 14:52:08.391386 1164 tablet_bootstrap.cc:492] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:08.397167 1164 tablet_bootstrap.cc:654] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.397969 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_38b74aa452d14ca5b22b7259f85af561" } modify_external_catalogs: true
I20250115 14:52:08.398219 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_38b74aa452d14ca5b22b7259f85af561" } modify_external_catalogs: true
I20250115 14:52:08.402268 452 catalog_manager.cc:5526] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:08.402817 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4ee8b63cc21145d8993b058d8c6d23c7 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]). Expected version 1 got 0
I20250115 14:52:08.403177 2325 tablet.cc:1721] T bb19d39258e444799d403a43475d9c8a P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:08.405064 2326 tablet.cc:1721] T bb19d39258e444799d403a43475d9c8a P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:08.405673 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a38d44069676412aa914ae1ff4ca0861" } modify_external_catalogs: true
I20250115 14:52:08.405927 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a38d44069676412aa914ae1ff4ca0861" } modify_external_catalogs: true
I20250115 14:52:08.407184 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0645565e1b904cff8d1ed879748d01bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.407708 1162 tablet_bootstrap.cc:492] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.407838 1162 ts_tablet_manager.cc:1397] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250115 14:52:08.408636 1162 raft_consensus.cc:357] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.408852 1162 raft_consensus.cc:383] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.408912 1162 raft_consensus.cc:738] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.408978 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1140016d11a74a2cadba494c0b319c54
I20250115 14:52:08.409155 1162 consensus_queue.cc:260] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.409533 455 catalog_manager.cc:6596] ReplaceTablet: tablet 1140016d11a74a2cadba494c0b319c54 deleted and replaced by tablet 068771fee92f403d8997646291a83dc6
I20250115 14:52:08.410110 1162 ts_tablet_manager.cc:1428] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:08.410367 1162 tablet_bootstrap.cc:492] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.410452 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 1140016d11a74a2cadba494c0b319c54 with delete_type TABLET_DATA_DELETED (replaced by tablet 068771fee92f403d8997646291a83dc6) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.411163 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 1140016d11a74a2cadba494c0b319c54 with delete_type TABLET_DATA_DELETED (replaced by tablet 068771fee92f403d8997646291a83dc6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.411597 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 1140016d11a74a2cadba494c0b319c54 with delete_type TABLET_DATA_DELETED (replaced by tablet 068771fee92f403d8997646291a83dc6) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.412989 1162 tablet_bootstrap.cc:654] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.362468 1167 rpcz_store.cc:276] 0115 14:52:07.094487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.095245 (+ 758us) service_pool.cc:225] Handling call
0115 14:52:08.293381 (+1198136us) ts_tablet_manager.cc:1145] Deleting tablet fc3d2ca7cd7c48a79ed1396fbbdc6866
0115 14:52:08.293391 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.303261 (+ 9870us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.339942 (+ 36681us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.362333 (+ 22391us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197986}
I20250115 14:52:08.414448 2329 tablet.cc:1721] T bb19d39258e444799d403a43475d9c8a P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:08.417536 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2d04209fd34918b83978325a5e1df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.417735 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet b5177265ba974ad6a381adb1e8c49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.417856 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1ce59e05054a009cd36e12db35d449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.417989 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 with delete_type TABLET_DATA_DELETED (replaced by tablet bc1440e3cd444cf8b3365a0af8de8958) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.417821 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_01c14574e8c94e2aac622d622f8ba52c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.418411 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01c14574e8c94e2aac622d622f8ba52c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.419039 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet fc3d2ca7cd7c48a79ed1396fbbdc6866 (table default.Table_e81cf878cdbf4e6f8fcb67db9a86a8af [id=3cd93abad1544a5a89086b8ead3a4df2]) successfully deleted
I20250115 14:52:08.423781 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 42bfa2dd86af4a29916081728ef71369 (DEFAULT_TABLE table=default.Table_9a47006d838d44b6ae411fddec170586 [id=c39d1ab3df524eb3bbbd22e42e626acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.424146 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bfa2dd86af4a29916081728ef71369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.425817 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 7830c683a109448ab06c15b5de692e8b (DEFAULT_TABLE table=default.Table_c5f3bc4addaa42d7acaf157e65707baa [id=c3e36b7d052844f8b092fdb6c87df49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.426165 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7830c683a109448ab06c15b5de692e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.426801 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 221b3271c2734828a96c4c86ab031dfc
I20250115 14:52:08.427488 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7654d2831f0141858b52379b767b3a70
W20250115 14:52:08.427978 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0645565e1b904cff8d1ed879748d01bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.428558 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c9a5593068d54d7bb3874cf434505e73 (DEFAULT_TABLE table=default.Table_a0070c4d1be947df84bf19c1efb88fa1 [id=84efaff9f0d44937a7379bf76905e838]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.428912 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a5593068d54d7bb3874cf434505e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.429385 1751 ts_tablet_manager.cc:1918] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.429468 1751 log.cc:1198] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/fa57c1fa96594a40b2551ad503127e4f
W20250115 14:52:08.429425 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1140016d11a74a2cadba494c0b319c54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1140016d11a74a2cadba494c0b319c54 already in progress: creating tablet
I20250115 14:52:08.429630 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 221b3271c2734828a96c4c86ab031dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.429791 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 7654d2831f0141858b52379b767b3a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.429994 748 tablet_service.cc:1467] Processing CreateTablet for tablet 42bfa2dd86af4a29916081728ef71369 (DEFAULT_TABLE table=default.Table_9a47006d838d44b6ae411fddec170586 [id=c39d1ab3df524eb3bbbd22e42e626acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.430357 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bfa2dd86af4a29916081728ef71369. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:08.430578 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 221b3271c2734828a96c4c86ab031dfc because the tablet was not found. No further retry: Not found: Tablet not found: 221b3271c2734828a96c4c86ab031dfc
W20250115 14:52:08.430814 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7654d2831f0141858b52379b767b3a70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7654d2831f0141858b52379b767b3a70 already in progress: creating tablet
I20250115 14:52:08.431330 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 7654d2831f0141858b52379b767b3a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.431497 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 221b3271c2734828a96c4c86ab031dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.432534 752 tablet_service.cc:1467] Processing CreateTablet for tablet 7830c683a109448ab06c15b5de692e8b (DEFAULT_TABLE table=default.Table_c5f3bc4addaa42d7acaf157e65707baa [id=c3e36b7d052844f8b092fdb6c87df49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.432866 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7830c683a109448ab06c15b5de692e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.435091 1751 ts_tablet_manager.cc:1939] T fa57c1fa96594a40b2551ad503127e4f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.435524 879 tablet_service.cc:1467] Processing CreateTablet for tablet 42bfa2dd86af4a29916081728ef71369 (DEFAULT_TABLE table=default.Table_9a47006d838d44b6ae411fddec170586 [id=c39d1ab3df524eb3bbbd22e42e626acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.435782 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 221b3271c2734828a96c4c86ab031dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.435801 741 tablet_service.cc:1467] Processing CreateTablet for tablet 7db35ff0383549de86230bb1eb89389f (DEFAULT_TABLE table=default.Table_f72ab5996cfd4471996f0b1034d3e422 [id=07e9a6f3d857429dac4ff951ad9e7aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.435902 1751 tablet_replica.cc:331] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.436096 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db35ff0383549de86230bb1eb89389f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.436111 886 tablet_service.cc:1467] Processing CreateTablet for tablet c9a5593068d54d7bb3874cf434505e73 (DEFAULT_TABLE table=default.Table_a0070c4d1be947df84bf19c1efb88fa1 [id=84efaff9f0d44937a7379bf76905e838]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.436532 893 tablet_service.cc:1467] Processing CreateTablet for tablet 7830c683a109448ab06c15b5de692e8b (DEFAULT_TABLE table=default.Table_c5f3bc4addaa42d7acaf157e65707baa [id=c3e36b7d052844f8b092fdb6c87df49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.436748 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 7654d2831f0141858b52379b767b3a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.437121 740 tablet_service.cc:1467] Processing CreateTablet for tablet c9a5593068d54d7bb3874cf434505e73 (DEFAULT_TABLE table=default.Table_a0070c4d1be947df84bf19c1efb88fa1 [id=84efaff9f0d44937a7379bf76905e838]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.437431 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a5593068d54d7bb3874cf434505e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.440604 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet fa57c1fa96594a40b2551ad503127e4f (table default.Table_958850c3f0a54c49a8dd42b180068897 [id=8242d7a08fcd4d4e836f6b70b1bacdc1]) successfully deleted
I20250115 14:52:08.441553 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_22a3195f3aa44230a56a246622e2d946"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.442167 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22a3195f3aa44230a56a246622e2d946 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.443220 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42bfa2dd86af4a29916081728ef71369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.443457 891 tablet_service.cc:1467] Processing CreateTablet for tablet 7db35ff0383549de86230bb1eb89389f (DEFAULT_TABLE table=default.Table_f72ab5996cfd4471996f0b1034d3e422 [id=07e9a6f3d857429dac4ff951ad9e7aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.445333 1751 raft_consensus.cc:2238] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.446008 1751 raft_consensus.cc:2267] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.446954 1751 ts_tablet_manager.cc:1905] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.447607 2340 raft_consensus.cc:491] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.447721 2340 raft_consensus.cc:513] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.448319 2340 leader_election.cc:290] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.449225 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f400880835ab4efdb2af0be5553d20dc
W20250115 14:52:08.449354 847 leader_election.cc:343] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e034d7761a5247c79414b7112a1c97b2
I20250115 14:52:08.450060 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e034d7761a5247c79414b7112a1c97b2" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.450580 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db35ff0383549de86230bb1eb89389f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:08.450551 848 leader_election.cc:343] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.450672 848 leader_election.cc:304] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.451015 2340 raft_consensus.cc:2743] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.451229 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f400880835ab4efdb2af0be5553d20dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:08.451897 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f400880835ab4efdb2af0be5553d20dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f400880835ab4efdb2af0be5553d20dc already in progress: creating tablet
I20250115 14:52:08.452457 1162 tablet_bootstrap.cc:492] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.452554 1162 ts_tablet_manager.cc:1397] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.004s
I20250115 14:52:08.453143 1162 raft_consensus.cc:357] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.453274 1162 raft_consensus.cc:383] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.453305 1162 raft_consensus.cc:738] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.453464 1162 consensus_queue.cc:260] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.453939 1162 ts_tablet_manager.cc:1428] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:08.454535 1168 tablet_bootstrap.cc:492] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:08.454655 1168 ts_tablet_manager.cc:1397] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.065s user 0.001s sys 0.004s
I20250115 14:52:08.455266 1168 raft_consensus.cc:357] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.455428 1168 raft_consensus.cc:383] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.455471 1168 raft_consensus.cc:738] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.456004 449 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 8561af93a9c247b981216586189221d1
I20250115 14:52:08.459246 1162 tablet_bootstrap.cc:492] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.461203 1162 tablet_bootstrap.cc:654] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.461850 1168 consensus_queue.cc:260] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.462993 1751 ts_tablet_manager.cc:1918] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.463086 1751 log.cc:1198] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9e95d09efd2e44cc81cdcbf87ee453fe
I20250115 14:52:08.463582 1168 ts_tablet_manager.cc:1428] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250115 14:52:08.463593 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8cb55ab9a193424ab91010e11b559f39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.463750 2340 raft_consensus.cc:491] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.463863 2340 raft_consensus.cc:513] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:08.464239 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cb55ab9a193424ab91010e11b559f39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.464442 2340 leader_election.cc:290] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.464813 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43605c995df4400e91139199c9514bfd" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.465018 771 raft_consensus.cc:2463] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
W20250115 14:52:08.465278 847 leader_election.cc:343] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 43605c995df4400e91139199c9514bfd
I20250115 14:52:08.465390 847 leader_election.cc:304] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:08.465616 2340 raft_consensus.cc:2798] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:08.465683 2340 raft_consensus.cc:491] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:08.465750 2340 raft_consensus.cc:3054] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.467221 451 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:08.467473 451 catalog_manager.cc:5526] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d7ecece2ab49e1be7079216517d13b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d2d7ecece2ab49e1be7079216517d13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:08.467844 2340 raft_consensus.cc:513] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.467949 451 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4ee8b63cc21145d8993b058d8c6d23c7 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]). Expected version 1 got 0
I20250115 14:52:08.468448 2340 leader_election.cc:290] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:08.469205 847 leader_election.cc:343] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 43605c995df4400e91139199c9514bfd
I20250115 14:52:08.469332 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43605c995df4400e91139199c9514bfd" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:08.469499 771 raft_consensus.cc:3054] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.469825 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a5593068d54d7bb3874cf434505e73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.471210 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet f400880835ab4efdb2af0be5553d20dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.471393 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 742792be08444792892bd9cd7336dda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.471911 771 raft_consensus.cc:2463] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:08.472419 848 leader_election.cc:304] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:08.472618 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fa193ea0c83741f9af1c5110e55af30a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.472805 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7830c683a109448ab06c15b5de692e8b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:08.473169 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa193ea0c83741f9af1c5110e55af30a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.473234 882 tablet_service.cc:1514] Processing DeleteTablet for tablet f400880835ab4efdb2af0be5553d20dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.473394 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 43605c995df4400e91139199c9514bfd with delete_type TABLET_DATA_DELETED (replaced by tablet e910c387e6a24caf95e14b57c8b83e6f) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.473513 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8efcf1a2db44469cd8dda0b59c34ba) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.473616 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e95d09efd2e44cc81cdcbf87ee453fe with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd94e4a7fef44f5be462e7a77e8a123) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.473727 2340 raft_consensus.cc:2798] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:08.473812 2340 raft_consensus.cc:695] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:08.474025 2340 consensus_queue.cc:237] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.477830 1168 tablet_bootstrap.cc:492] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.479355 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9b57b8b9434f55ace5a4f47380df54 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8bae9867d745afb1dfdf47bcb881da) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.480183 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_607f5ab1bfce4111821d459c200808f7" } new_table_name: "default.Table_572fd8a32b744555a1e45e1321f35352" modify_external_catalogs: true
I20250115 14:52:08.482834 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_18573a0a9c86499d9223ac71aa866a24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.483561 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18573a0a9c86499d9223ac71aa866a24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.484320 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet be2fd31f74a94787a5e8ccae3c5ef245 (table default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec [id=724c015c8f3f4aa581ac6faa84187b06]). Expected version 1 got 0
I20250115 14:52:08.485180 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5ea7727ee96e426594a8c8af14d9545d" } new_table_name: "default.Table_f61f66337491436e90983173399d81ae" modify_external_catalogs: true
I20250115 14:52:08.491799 2341 raft_consensus.cc:491] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.491943 2341 raft_consensus.cc:513] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.492588 2341 leader_election.cc:290] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.492851 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8442dc919d5f4d63b9b2ce7276cc0bc5" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.493438 848 leader_election.cc:343] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.494974 1164 tablet_bootstrap.cc:492] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:08.495107 1164 ts_tablet_manager.cc:1397] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.002s
I20250115 14:52:08.495822 1164 raft_consensus.cc:357] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.496001 1164 raft_consensus.cc:383] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.496047 1164 raft_consensus.cc:738] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.496511 1168 tablet_bootstrap.cc:654] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.496526 1164 consensus_queue.cc:260] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.499337 1164 ts_tablet_manager.cc:1428] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:08.500581 1164 tablet_bootstrap.cc:492] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:08.502049 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69c0368e67624a6e8d1f6a4a6275466b consensus state has no leader
I20250115 14:52:08.503222 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_241b0cb6766b476da1e9e3a989f69e8d [id=13c49e5a84b94acca8344f3e04e99736] alter complete (version 1)
I20250115 14:52:08.503458 1164 tablet_bootstrap.cc:654] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.505523 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 814) took 1411 ms. Trace:
I20250115 14:52:08.506995 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 7db35ff0383549de86230bb1eb89389f (DEFAULT_TABLE table=default.Table_f72ab5996cfd4471996f0b1034d3e422 [id=07e9a6f3d857429dac4ff951ad9e7aea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.507354 1167 rpcz_store.cc:276] 0115 14:52:07.094404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.153685 (+ 59281us) service_pool.cc:225] Handling call
0115 14:52:08.414573 (+1260888us) ts_tablet_manager.cc:1145] Deleting tablet ded663ac1197454bbde3c4c2d526002d
0115 14:52:08.442783 (+ 28210us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.505498 (+ 62715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254630}
W20250115 14:52:08.507961 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ded663ac1197454bbde3c4c2d526002d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ded663ac1197454bbde3c4c2d526002d already in progress: creating tablet
I20250115 14:52:08.508489 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 827) took 1290 ms. Trace:
I20250115 14:52:08.515589 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 02ce16fea4f9469c9fc06fa24f6554be from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:08.515956 1751 ts_tablet_manager.cc:1939] T 9e95d09efd2e44cc81cdcbf87ee453fe P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.516331 752 tablet_service.cc:1467] Processing CreateTablet for tablet daacfd23ec90436089382270e45f9e9b (DEFAULT_TABLE table=default.Table_043c9ff0bc444163a82cfc7c4d02c35f [id=fcfe896489614e25b04c37914689f089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.516580 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 43605c995df4400e91139199c9514bfd with delete_type TABLET_DATA_DELETED (replaced by tablet e910c387e6a24caf95e14b57c8b83e6f) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.516995 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daacfd23ec90436089382270e45f9e9b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:08.517982 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e185963130bf49a1af88ab9e14e8e1bc consensus state has no leader
I20250115 14:52:08.518128 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1032bbe4a65c4fa1b238c728e1ab2fdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.521839 1162 tablet_bootstrap.cc:492] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.521978 1162 ts_tablet_manager.cc:1397] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.002s
I20250115 14:52:08.522696 1162 raft_consensus.cc:357] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.522915 1162 raft_consensus.cc:383] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.522969 1162 raft_consensus.cc:738] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.523161 1162 consensus_queue.cc:260] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.524003 1162 ts_tablet_manager.cc:1428] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:08.524585 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db35ff0383549de86230bb1eb89389f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.524680 1166 tablet_replica.cc:331] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:08.524865 1166 raft_consensus.cc:2238] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.525020 1166 raft_consensus.cc:2267] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.525730 1166 ts_tablet_manager.cc:1905] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.528568 1162 tablet_bootstrap.cc:492] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:08.528563 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1032bbe4a65c4fa1b238c728e1ab2fdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.530198 1162 tablet_bootstrap.cc:654] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.532212 1751 tablet_replica.cc:331] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.532418 1751 raft_consensus.cc:2238] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.532543 1751 raft_consensus.cc:2267] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.533064 1751 ts_tablet_manager.cc:1905] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.533288 2273 raft_consensus.cc:491] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.533398 2273 raft_consensus.cc:513] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.533975 2273 leader_election.cc:290] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.535208 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95718d39a7824517a36c49c3b86fef30" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.535777 984 leader_election.cc:343] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.537886 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9e95d09efd2e44cc81cdcbf87ee453fe (table default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]) successfully deleted
I20250115 14:52:08.538914 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2989f3ca246d42528088a1e17c79eb91" } modify_external_catalogs: true
I20250115 14:52:08.539116 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2989f3ca246d42528088a1e17c79eb91" } modify_external_catalogs: true
I20250115 14:52:08.540486 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fd29d5224a264121b8da1f6186711fd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.540969 1166 ts_tablet_manager.cc:1918] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:08.541039 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd29d5224a264121b8da1f6186711fd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.541070 1166 log.cc:1198] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/43605c995df4400e91139199c9514bfd
I20250115 14:52:08.541354 1166 ts_tablet_manager.cc:1939] T 43605c995df4400e91139199c9514bfd P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:08.542454 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_272f9650cb3142f781f1d4d7c4769dfe" } new_table_name: "default.Table_3e67bbd220754aac89a7b142edcb206c" modify_external_catalogs: true
I20250115 14:52:08.545346 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dfd7391e00dc4815aedb1a4c64eba86f" } new_table_name: "default.Table_54e0718fb03d45629ac03649aacf44ba" modify_external_catalogs: true
I20250115 14:52:08.548563 1751 ts_tablet_manager.cc:1918] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.548975 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cc65d5ae73344c87b1bcf475ebac6b00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.549535 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc65d5ae73344c87b1bcf475ebac6b00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.553298 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 43605c995df4400e91139199c9514bfd (table default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]) successfully deleted
I20250115 14:52:08.554931 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet ca128bf3d6fc430298da932a78f54184 (table default.Table_8f53bbc08e5340aab38bb0ce8ef7aa16 [id=6cdcb9c1b34642d8a792807746ac26f5]). Expected version 1 got 0
I20250115 14:52:08.555244 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet be2fd31f74a94787a5e8ccae3c5ef245 (table default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec [id=724c015c8f3f4aa581ac6faa84187b06]). Expected version 1 got 0
I20250115 14:52:08.559931 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ed3d35aa7cd046848bbe7989fdfbdae5" } new_table_name: "default.Table_dbb09336b4004c35b3ee4880598b02d8" modify_external_catalogs: true
I20250115 14:52:08.560539 2324 raft_consensus.cc:491] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.560701 2324 raft_consensus.cc:513] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.561270 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 84844b5f50af408ab7dda73f7421cd9d
I20250115 14:52:08.561594 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a0d97fa57484b9d8cb879fedda61612 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.520221 2341 raft_consensus.cc:491] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.561827 2341 raft_consensus.cc:513] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.562431 2341 leader_election.cc:290] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.521131 1168 tablet_bootstrap.cc:492] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
W20250115 14:52:08.524482 847 leader_election.cc:343] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 8442dc919d5f4d63b9b2ce7276cc0bc5
I20250115 14:52:08.562594 847 leader_election.cc:304] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.534601 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95718d39a7824517a36c49c3b86fef30" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.562870 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2fd31f74a94787a5e8ccae3c5ef245" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.563330 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 84844b5f50af408ab7dda73f7421cd9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:08.563551 848 leader_election.cc:343] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.561535 2138 raft_consensus.cc:491] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.564388 2138 raft_consensus.cc:513] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.564567 1168 ts_tablet_manager.cc:1397] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.087s user 0.004s sys 0.000s
I20250115 14:52:08.564805 1843 raft_consensus.cc:2743] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.564946 2138 leader_election.cc:290] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.565270 1168 raft_consensus.cc:357] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.565425 1168 raft_consensus.cc:383] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.565474 1168 raft_consensus.cc:738] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.565475 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0d97fa57484b9d8cb879fedda61612" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.565652 771 raft_consensus.cc:2463] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:08.565816 1168 consensus_queue.cc:260] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.566076 848 leader_election.cc:304] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:08.566426 1168 ts_tablet_manager.cc:1428] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:08.568102 2341 raft_consensus.cc:2798] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:08.568174 2341 raft_consensus.cc:491] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:08.568228 2341 raft_consensus.cc:3054] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.568445 1166 tablet_replica.cc:331] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:08.568702 1166 raft_consensus.cc:2238] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.568892 1166 raft_consensus.cc:2267] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.569013 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 84844b5f50af408ab7dda73f7421cd9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.569142 1167 rpcz_store.cc:276] 0115 14:52:07.217958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.218016 (+ 58us) service_pool.cc:225] Handling call
0115 14:52:08.508361 (+1290345us) ts_tablet_manager.cc:1145] Deleting tablet 79736602ea2646b39708ca9c6834e847
0115 14:52:08.508371 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.508470 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290179}
I20250115 14:52:08.569505 1166 ts_tablet_manager.cc:1905] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.570994 2341 raft_consensus.cc:513] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.571317 2324 leader_election.cc:290] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.571511 2341 leader_election.cc:290] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:08.573426 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 79736602ea2646b39708ca9c6834e847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79736602ea2646b39708ca9c6834e847 already in progress: creating tablet
W20250115 14:52:08.573658 847 leader_election.cc:343] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 3a0d97fa57484b9d8cb879fedda61612
W20250115 14:52:08.573854 847 leader_election.cc:343] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 3a0d97fa57484b9d8cb879fedda61612
I20250115 14:52:08.574901 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a0d97fa57484b9d8cb879fedda61612" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:08.575250 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3592cd57b6b945288efcb39b7753ab52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.575496 848 leader_election.cc:343] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: Tablet not RUNNING: STOPPED
I20250115 14:52:08.575592 848 leader_election.cc:304] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:08.575829 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3592cd57b6b945288efcb39b7753ab52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.576143 2340 raft_consensus.cc:2743] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.579100 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:08.579973 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a8000f5474f62aae52c8e88e45276" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.580188 909 raft_consensus.cc:2463] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:08.580576 886 tablet_service.cc:1467] Processing CreateTablet for tablet daacfd23ec90436089382270e45f9e9b (DEFAULT_TABLE table=default.Table_043c9ff0bc444163a82cfc7c4d02c35f [id=fcfe896489614e25b04c37914689f089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.581065 710 leader_election.cc:304] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:08.581343 2324 raft_consensus.cc:2798] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:08.581413 2324 raft_consensus.cc:491] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:08.581470 2324 raft_consensus.cc:3054] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.581876 1168 tablet_bootstrap.cc:492] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.583802 2324 raft_consensus.cc:513] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.584095 879 tablet_service.cc:1467] Processing CreateTablet for tablet 5f09898e70cd4ef2b05ea3f567a3137d (DEFAULT_TABLE table=default.Table_888f5c5df23b4835a34d43300e9f3806 [id=a6b71b7786504f61bfd489af48b88b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.584859 2324 leader_election.cc:290] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.585621 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a8000f5474f62aae52c8e88e45276" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:08.585839 909 raft_consensus.cc:3054] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.564153 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2fd31f74a94787a5e8ccae3c5ef245" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:52:08.564198 983 leader_election.cc:343] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.571777 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 84844b5f50af408ab7dda73f7421cd9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.588842 909 raft_consensus.cc:2463] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:08.588891 1048 raft_consensus.cc:2463] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:08.582443 1028 tablet_service.cc:1467] Processing CreateTablet for tablet daacfd23ec90436089382270e45f9e9b (DEFAULT_TABLE table=default.Table_043c9ff0bc444163a82cfc7c4d02c35f [id=fcfe896489614e25b04c37914689f089]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.585690 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 5f09898e70cd4ef2b05ea3f567a3137d (DEFAULT_TABLE table=default.Table_888f5c5df23b4835a34d43300e9f3806 [id=a6b71b7786504f61bfd489af48b88b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.589313 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daacfd23ec90436089382270e45f9e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.589325 893 tablet_service.cc:1467] Processing CreateTablet for tablet 16a88475f9ff42c1952e5a14ec852e90 (DEFAULT_TABLE table=default.Table_298f14ed9b9b41fc802a48c4f1bbe07f [id=3c1f2f64fe374b5294ef6130f252840f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.589505 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f09898e70cd4ef2b05ea3f567a3137d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.590837 1168 tablet_bootstrap.cc:654] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.591634 1751 log.cc:1198] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a600fa0f0ec049dca2803bf7d2c74a95
I20250115 14:52:08.591967 1751 ts_tablet_manager.cc:1939] T a600fa0f0ec049dca2803bf7d2c74a95 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.588891 983 leader_election.cc:304] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.593046 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a8000f5474f62aae52c8e88e45276" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:08.593148 2273 raft_consensus.cc:2743] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.593171 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2a8000f5474f62aae52c8e88e45276" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:08.593225 1048 raft_consensus.cc:2463] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:08.593309 1047 raft_consensus.cc:3054] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.593377 1751 tablet_replica.cc:331] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.594190 847 leader_election.cc:304] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.594461 2341 raft_consensus.cc:2798] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:08.594534 2341 raft_consensus.cc:491] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:08.594586 2341 raft_consensus.cc:3054] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.594825 710 leader_election.cc:304] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:08.595572 1047 raft_consensus.cc:2463] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:08.596051 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a600fa0f0ec049dca2803bf7d2c74a95 (table default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d [id=ab53ab1ed1b34dafa524ccb35291fd62]) successfully deleted
I20250115 14:52:08.596729 2324 raft_consensus.cc:2798] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:08.597313 2324 raft_consensus.cc:695] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:08.597584 2341 raft_consensus.cc:513] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.597676 2324 consensus_queue.cc:237] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.598462 2341 leader_election.cc:290] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:08.599324 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2fd31f74a94787a5e8ccae3c5ef245" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:08.599504 1047 raft_consensus.cc:3054] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.600721 752 tablet_service.cc:1467] Processing CreateTablet for tablet 5f09898e70cd4ef2b05ea3f567a3137d (DEFAULT_TABLE table=default.Table_888f5c5df23b4835a34d43300e9f3806 [id=a6b71b7786504f61bfd489af48b88b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.601439 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f09898e70cd4ef2b05ea3f567a3137d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.601805 1047 raft_consensus.cc:2463] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:08.602663 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be2fd31f74a94787a5e8ccae3c5ef245" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:08.603070 847 leader_election.cc:304] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:08.603677 2138 raft_consensus.cc:2798] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:08.603813 2138 raft_consensus.cc:695] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:08.604039 2138 consensus_queue.cc:237] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:08.605371 848 leader_election.cc:343] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.607960 1162 tablet_bootstrap.cc:492] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.608095 1162 ts_tablet_manager.cc:1397] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.080s user 0.001s sys 0.001s
I20250115 14:52:08.608968 1162 raft_consensus.cc:357] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.609229 1162 raft_consensus.cc:383] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.609282 1162 raft_consensus.cc:738] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.609483 1162 consensus_queue.cc:260] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.609946 1162 ts_tablet_manager.cc:1428] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:08.610200 1162 tablet_bootstrap.cc:492] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.611874 1162 tablet_bootstrap.cc:654] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.614056 740 tablet_service.cc:1467] Processing CreateTablet for tablet 16a88475f9ff42c1952e5a14ec852e90 (DEFAULT_TABLE table=default.Table_298f14ed9b9b41fc802a48c4f1bbe07f [id=3c1f2f64fe374b5294ef6130f252840f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.614457 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a88475f9ff42c1952e5a14ec852e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.616130 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 828) took 1396 ms. Trace:
I20250115 14:52:08.616214 1167 rpcz_store.cc:276] 0115 14:52:07.219232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.219278 (+ 46us) service_pool.cc:225] Handling call
0115 14:52:08.569388 (+1350110us) ts_tablet_manager.cc:1145] Deleting tablet 40eae66b35974524992b2ef943273291
0115 14:52:08.615995 (+ 46607us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.616107 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1349968}
W20250115 14:52:08.616719 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 40eae66b35974524992b2ef943273291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40eae66b35974524992b2ef943273291 already in progress: creating tablet
I20250115 14:52:08.617053 1167 tablet_replica.cc:331] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:08.617280 1167 raft_consensus.cc:2238] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.617457 1167 raft_consensus.cc:2267] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.617981 1167 ts_tablet_manager.cc:1905] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.621780 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daacfd23ec90436089382270e45f9e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.622187 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a88475f9ff42c1952e5a14ec852e90. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:08.629232 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1218f0bdb1e14bd78d76c6a06805f38c consensus state has no leader
W20250115 14:52:08.634524 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a1a69b6de5d433f82579b1259548fd2 consensus state has no leader
I20250115 14:52:08.635277 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_69711e6d76ee4d289ccb17b881ec97bf" } modify_external_catalogs: true
I20250115 14:52:08.635463 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_69711e6d76ee4d289ccb17b881ec97bf" } modify_external_catalogs: true
I20250115 14:52:08.635668 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f09898e70cd4ef2b05ea3f567a3137d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.637157 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0636a1492a2542d99c65b5d1436dd981" } modify_external_catalogs: true
I20250115 14:52:08.637321 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0636a1492a2542d99c65b5d1436dd981" } modify_external_catalogs: true
I20250115 14:52:08.638612 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
W20250115 14:52:08.640214 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0f82d944a7e04ace8e4addb3897d2c8e consensus state has no leader
I20250115 14:52:08.643247 2241 raft_consensus.cc:491] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.645274 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 32cb8768ca424aea87682295f30830a9
I20250115 14:52:08.647214 891 tablet_service.cc:1467] Processing CreateTablet for tablet 64a76d2772534687b1391e45b9cb984b (DEFAULT_TABLE table=default.Table_225689bca59c415d884b3d3ca93d556b [id=3b0cff3ca69f43388447ae21f6de28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.647559 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64a76d2772534687b1391e45b9cb984b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.647485 886 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce37631807c464484e19a3801441674 (DEFAULT_TABLE table=default.Table_18716bd80651469abf46d1c1e1a8729a [id=db439563b669420c8793034d7c172fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.647842 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce37631807c464484e19a3801441674. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.643390 2241 raft_consensus.cc:513] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.650231 2241 leader_election.cc:290] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.650561 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce37631807c464484e19a3801441674 (DEFAULT_TABLE table=default.Table_18716bd80651469abf46d1c1e1a8729a [id=db439563b669420c8793034d7c172fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.650727 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa163b6f9e846c6858892cd61a3e682 (DEFAULT_TABLE table=default.Table_d25c13a8ea78433ea4d007bbc8a846a7 [id=29022fe8f8294047a6124970e28a40e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.650938 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 16a88475f9ff42c1952e5a14ec852e90 (DEFAULT_TABLE table=default.Table_298f14ed9b9b41fc802a48c4f1bbe07f [id=3c1f2f64fe374b5294ef6130f252840f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.651466 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37f4c2ab9d248a1af817cfa2c2e2ff0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:08.651980 983 leader_election.cc:343] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.652063 740 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce37631807c464484e19a3801441674 (DEFAULT_TABLE table=default.Table_18716bd80651469abf46d1c1e1a8729a [id=db439563b669420c8793034d7c172fed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.652588 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce37631807c464484e19a3801441674. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.652526 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37f4c2ab9d248a1af817cfa2c2e2ff0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.653004 984 leader_election.cc:343] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.653095 984 leader_election.cc:304] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.654217 2241 raft_consensus.cc:2743] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.656930 1162 tablet_bootstrap.cc:492] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.657074 1162 ts_tablet_manager.cc:1397] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20250115 14:52:08.657588 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cb8768ca424aea87682295f30830a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.657889 1168 tablet_bootstrap.cc:492] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:08.657807 1162 raft_consensus.cc:357] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.657996 1162 raft_consensus.cc:383] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.658017 1168 ts_tablet_manager.cc:1397] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.001s
I20250115 14:52:08.658056 1162 raft_consensus.cc:738] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.658802 1168 raft_consensus.cc:357] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.658973 1168 raft_consensus.cc:383] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.659029 1168 raft_consensus.cc:738] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.659240 1168 consensus_queue.cc:260] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.647027 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa163b6f9e846c6858892cd61a3e682 (DEFAULT_TABLE table=default.Table_d25c13a8ea78433ea4d007bbc8a846a7 [id=29022fe8f8294047a6124970e28a40e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.660118 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa163b6f9e846c6858892cd61a3e682. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.660145 1168 ts_tablet_manager.cc:1428] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:08.660441 1168 tablet_bootstrap.cc:492] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.664440 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cb8768ca424aea87682295f30830a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.667302 1162 consensus_queue.cc:260] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.667872 1162 ts_tablet_manager.cc:1428] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250115 14:52:08.668424 1168 tablet_bootstrap.cc:654] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.675838 1167 ts_tablet_manager.cc:1918] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.676342 1167 log.cc:1198] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/0e924821d12045f6bc658461675aead4
I20250115 14:52:08.676646 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 64a76d2772534687b1391e45b9cb984b (DEFAULT_TABLE table=default.Table_225689bca59c415d884b3d3ca93d556b [id=3b0cff3ca69f43388447ae21f6de28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.676882 1167 ts_tablet_manager.cc:1939] T 0e924821d12045f6bc658461675aead4 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:08.676949 748 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa163b6f9e846c6858892cd61a3e682 (DEFAULT_TABLE table=default.Table_d25c13a8ea78433ea4d007bbc8a846a7 [id=29022fe8f8294047a6124970e28a40e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.677322 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa163b6f9e846c6858892cd61a3e682. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.677570 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 829) took 1438 ms. Trace:
I20250115 14:52:08.677644 1167 rpcz_store.cc:276] 0115 14:52:07.239311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.239365 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:08.617006 (+1377641us) ts_tablet_manager.cc:1145] Deleting tablet 0e924821d12045f6bc658461675aead4
0115 14:52:08.617016 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.665806 (+ 48790us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.675737 (+ 9931us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.677554 (+ 1817us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1377423}
I20250115 14:52:08.678027 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 830) took 1438 ms. Trace:
I20250115 14:52:08.678094 1167 rpcz_store.cc:276] 0115 14:52:07.239523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.239609 (+ 86us) service_pool.cc:225] Handling call
0115 14:52:08.677911 (+1438302us) ts_tablet_manager.cc:1145] Deleting tablet 7ff71ded3b074ff0979e2090131f4eaa
0115 14:52:08.677920 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.678012 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438152}
I20250115 14:52:08.678385 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 831) took 1425 ms. Trace:
I20250115 14:52:08.678447 1167 rpcz_store.cc:276] 0115 14:52:07.252536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.252590 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:08.678290 (+1425700us) ts_tablet_manager.cc:1145] Deleting tablet 338706b2d7514f94afd3579da458b4ec
0115 14:52:08.678298 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.678371 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1425497}
I20250115 14:52:08.678496 1018 tablet_service.cc:1467] Processing CreateTablet for tablet c7f0084c0e0146fa84bb332ffecdc343 (DEFAULT_TABLE table=default.Table_9a5f72bcad5e4bfe9b0e4ee3eeffd7f4 [id=f2e5424ef5df44ebacdc012ac0f55d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.678709 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 832) took 1425 ms. Trace:
I20250115 14:52:08.678786 1167 rpcz_store.cc:276] 0115 14:52:07.253257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.253340 (+ 83us) service_pool.cc:225] Handling call
0115 14:52:08.678620 (+1425280us) ts_tablet_manager.cc:1145] Deleting tablet b579be5c683e4fd8a5d5484fdb89250c
0115 14:52:08.678628 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.678695 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1425111}
I20250115 14:52:08.679104 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 845) took 1292 ms. Trace:
I20250115 14:52:08.679170 1167 rpcz_store.cc:276] 0115 14:52:07.386715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.386765 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:08.678999 (+1292234us) ts_tablet_manager.cc:1145] Deleting tablet 6e2d55a4e32448fbb98f704dbf1657b9
0115 14:52:08.679010 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.679089 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292059}
I20250115 14:52:08.680428 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 221b3271c2734828a96c4c86ab031dfc (DEFAULT_TABLE table=default.Table_ea36c03e003e440d99efe3c4c33838fe [id=11553e13f9914ddab5608ec6e2752db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.681262 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cb8768ca424aea87682295f30830a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.682257 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 2087b4ce466f4167b7c32c7074dcf6e7 (DEFAULT_TABLE table=default.Table_32080fc25c67476db569c3357f008e2c [id=9b453c31d5cd4e6a805d95bd66faa8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.682729 752 tablet_service.cc:1467] Processing CreateTablet for tablet 64a76d2772534687b1391e45b9cb984b (DEFAULT_TABLE table=default.Table_225689bca59c415d884b3d3ca93d556b [id=3b0cff3ca69f43388447ae21f6de28f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.683337 886 tablet_service.cc:1467] Processing CreateTablet for tablet c7f0084c0e0146fa84bb332ffecdc343 (DEFAULT_TABLE table=default.Table_9a5f72bcad5e4bfe9b0e4ee3eeffd7f4 [id=f2e5424ef5df44ebacdc012ac0f55d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.683554 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 0e924821d12045f6bc658461675aead4 (table default.Table_c156b9383c644939a6ada0bb7d7cf235 [id=d1b6400bffad453ab3b0b85e55091aa8]) successfully deleted
I20250115 14:52:08.683815 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f0084c0e0146fa84bb332ffecdc343. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:08.683848 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7ff71ded3b074ff0979e2090131f4eaa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ff71ded3b074ff0979e2090131f4eaa already in progress: creating tablet
W20250115 14:52:08.684072 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 338706b2d7514f94afd3579da458b4ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 338706b2d7514f94afd3579da458b4ec already in progress: creating tablet
W20250115 14:52:08.684304 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b579be5c683e4fd8a5d5484fdb89250c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b579be5c683e4fd8a5d5484fdb89250c already in progress: creating tablet
W20250115 14:52:08.684541 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6e2d55a4e32448fbb98f704dbf1657b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2d55a4e32448fbb98f704dbf1657b9 already in progress: creating tablet
I20250115 14:52:08.684345 749 tablet_service.cc:1467] Processing CreateTablet for tablet c7f0084c0e0146fa84bb332ffecdc343 (DEFAULT_TABLE table=default.Table_9a5f72bcad5e4bfe9b0e4ee3eeffd7f4 [id=f2e5424ef5df44ebacdc012ac0f55d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.686272 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64a76d2772534687b1391e45b9cb984b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.686991 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f0084c0e0146fa84bb332ffecdc343. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.687718 879 tablet_service.cc:1467] Processing CreateTablet for tablet 2087b4ce466f4167b7c32c7074dcf6e7 (DEFAULT_TABLE table=default.Table_32080fc25c67476db569c3357f008e2c [id=9b453c31d5cd4e6a805d95bd66faa8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.688634 891 tablet_service.cc:1467] Processing CreateTablet for tablet 98137e1c6ffa4352af8abca6f8d5748d (DEFAULT_TABLE table=default.Table_f6993d1e422647d59233c375c8894af3 [id=aee53ca6780f4992996e152bf26ba4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.688210 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2087b4ce466f4167b7c32c7074dcf6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.689878 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98137e1c6ffa4352af8abca6f8d5748d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.691905 1751 raft_consensus.cc:2238] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.691874 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 98137e1c6ffa4352af8abca6f8d5748d (DEFAULT_TABLE table=default.Table_f6993d1e422647d59233c375c8894af3 [id=aee53ca6780f4992996e152bf26ba4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.692088 1751 raft_consensus.cc:2267] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.692778 909 raft_consensus.cc:3054] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:08.694835 1047 raft_consensus.cc:1270] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer d2d7ecece2ab49e1be7079216517d13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:08.695888 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98137e1c6ffa4352af8abca6f8d5748d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.696359 909 raft_consensus.cc:1270] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer d2d7ecece2ab49e1be7079216517d13b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:08.697540 876 tablet_service.cc:1467] Processing CreateTablet for tablet 3f99d417ef074f978a91236d31552315 (DEFAULT_TABLE table=default.Table_044eb9404b5446298ecd6e8f48215b59 [id=22ec9e3bab0b4e07b65749816b82dc48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.697902 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f99d417ef074f978a91236d31552315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.700588 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 3f99d417ef074f978a91236d31552315 (DEFAULT_TABLE table=default.Table_044eb9404b5446298ecd6e8f48215b59 [id=22ec9e3bab0b4e07b65749816b82dc48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.700989 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f99d417ef074f978a91236d31552315. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.701848 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 26d7f86ae5e547ecbfece0cb23d7c656 (DEFAULT_TABLE table=default.Table_e01109831b5f43a7b10352cdb82cc440 [id=5bf10dfd595f48ee9bfee83a865fb314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.702205 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d7f86ae5e547ecbfece0cb23d7c656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.702162 2322 consensus_queue.cc:1035] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:08.704459 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce37631807c464484e19a3801441674. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.704694 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 570a78d2b40246ac8f460dcca569ec9d (DEFAULT_TABLE table=default.Table_9e22bc322f6e4b8bbcbdb9e756b33180 [id=6378f2ec20c745b7a55607adc9ec3b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.704984 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570a78d2b40246ac8f460dcca569ec9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.707726 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa163b6f9e846c6858892cd61a3e682. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.709066 885 tablet_service.cc:1467] Processing CreateTablet for tablet 26d7f86ae5e547ecbfece0cb23d7c656 (DEFAULT_TABLE table=default.Table_e01109831b5f43a7b10352cdb82cc440 [id=5bf10dfd595f48ee9bfee83a865fb314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.709468 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d7f86ae5e547ecbfece0cb23d7c656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.688613 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 846) took 1293 ms. Trace:
I20250115 14:52:08.709896 1167 rpcz_store.cc:276] 0115 14:52:07.395337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.395385 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:08.688434 (+1293049us) ts_tablet_manager.cc:1145] Deleting tablet b5937d6ad0494734a402fdc594eb9415
0115 14:52:08.688442 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.688577 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1283811}
I20250115 14:52:08.710992 1014 tablet_service.cc:1467] Processing CreateTablet for tablet cb866ca7cf2d4d39a46cae8532ed0418 (DEFAULT_TABLE table=default.Table_cb164ba6d8f74120ae308a71c1a936db [id=43cbf44ddd7f41b7909fcab2320002ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.711355 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb866ca7cf2d4d39a46cae8532ed0418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.713315 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16a88475f9ff42c1952e5a14ec852e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.717684 1751 ts_tablet_manager.cc:1905] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.717135 880 tablet_service.cc:1467] Processing CreateTablet for tablet 570a78d2b40246ac8f460dcca569ec9d (DEFAULT_TABLE table=default.Table_9e22bc322f6e4b8bbcbdb9e756b33180 [id=6378f2ec20c745b7a55607adc9ec3b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.718508 882 tablet_service.cc:1467] Processing CreateTablet for tablet cb866ca7cf2d4d39a46cae8532ed0418 (DEFAULT_TABLE table=default.Table_cb164ba6d8f74120ae308a71c1a936db [id=43cbf44ddd7f41b7909fcab2320002ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.721892 1168 tablet_bootstrap.cc:492] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
W20250115 14:52:08.723610 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b5937d6ad0494734a402fdc594eb9415 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5937d6ad0494734a402fdc594eb9415 already in progress: creating tablet
I20250115 14:52:08.723692 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64a76d2772534687b1391e45b9cb984b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.724079 1162 tablet_bootstrap.cc:492] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.725224 1751 ts_tablet_manager.cc:1918] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.725329 1751 log.cc:1198] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b2a11b7b0dd6494daf7b017edd4317a5
I20250115 14:52:08.725807 1751 ts_tablet_manager.cc:1939] T b2a11b7b0dd6494daf7b017edd4317a5 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.726490 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 884) took 1132 ms. Trace:
I20250115 14:52:08.726580 1751 rpcz_store.cc:276] 0115 14:52:07.594228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.594279 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:08.592870 (+998591us) ts_tablet_manager.cc:1145] Deleting tablet b2a11b7b0dd6494daf7b017edd4317a5
0115 14:52:08.592879 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.720270 (+127391us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.725150 (+ 4880us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.726473 (+ 1323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998401}
I20250115 14:52:08.726961 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 885) took 1116 ms. Trace:
I20250115 14:52:08.726992 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b2a11b7b0dd6494daf7b017edd4317a5 (table default.Table_8e068b869e304185abaccf2d63685cd3 [id=5a50430da9af480282aa31475f196737]) successfully deleted
I20250115 14:52:08.727031 1751 rpcz_store.cc:276] 0115 14:52:07.610599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.610649 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:08.726851 (+1116202us) ts_tablet_manager.cc:1145] Deleting tablet e3c54fdc672f4709a44b2542212dfb5e
0115 14:52:08.726860 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.726949 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116062}
I20250115 14:52:08.727308 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 886) took 1069 ms. Trace:
I20250115 14:52:08.727370 1751 rpcz_store.cc:276] 0115 14:52:07.658032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.658078 (+ 46us) service_pool.cc:225] Handling call
0115 14:52:08.727225 (+1069147us) ts_tablet_manager.cc:1145] Deleting tablet 96349985df1e49e9b942eb610ccf70f9
0115 14:52:08.727233 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.727297 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068992}
W20250115 14:52:08.727473 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e3c54fdc672f4709a44b2542212dfb5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3c54fdc672f4709a44b2542212dfb5e already in progress: creating tablet
I20250115 14:52:08.727579 1751 tablet_replica.cc:331] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:08.727955 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 96349985df1e49e9b942eb610ccf70f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96349985df1e49e9b942eb610ccf70f9 already in progress: creating tablet
I20250115 14:52:08.728479 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f0084c0e0146fa84bb332ffecdc343. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.727761 1751 raft_consensus.cc:2238] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.729522 1751 raft_consensus.cc:2267] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.730054 1751 ts_tablet_manager.cc:1905] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.731783 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221b3271c2734828a96c4c86ab031dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.735450 1751 ts_tablet_manager.cc:1918] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:08.735548 1751 log.cc:1198] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/80c75a77d7154b5cb05a5d9ac8a20787
I20250115 14:52:08.735862 1751 ts_tablet_manager.cc:1939] T 80c75a77d7154b5cb05a5d9ac8a20787 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.736475 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2087b4ce466f4167b7c32c7074dcf6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.741191 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 887) took 1082 ms. Trace:
I20250115 14:52:08.741291 1751 rpcz_store.cc:276] 0115 14:52:07.658497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.658540 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:08.727543 (+1069003us) ts_tablet_manager.cc:1145] Deleting tablet 80c75a77d7154b5cb05a5d9ac8a20787
0115 14:52:08.727550 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.732926 (+ 5376us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.735378 (+ 2452us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.741170 (+ 5792us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1068858}
I20250115 14:52:08.741828 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]) successfully deleted
I20250115 14:52:08.743307 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 888) took 1083 ms. Trace:
I20250115 14:52:08.743388 1751 rpcz_store.cc:276] 0115 14:52:07.659361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.659403 (+ 42us) service_pool.cc:225] Handling call
0115 14:52:08.743178 (+1083775us) ts_tablet_manager.cc:1145] Deleting tablet ba21cf54bc074d95b456fd70a9286001
0115 14:52:08.743188 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.743288 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083664}
I20250115 14:52:08.744374 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 889) took 1082 ms. Trace:
I20250115 14:52:08.744458 1751 rpcz_store.cc:276] 0115 14:52:07.662221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.662264 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:08.744209 (+1081945us) ts_tablet_manager.cc:1145] Deleting tablet b0f33b5c22ee4f8bb3ee2d20c188ade0
0115 14:52:08.744218 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.744356 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1081766}
I20250115 14:52:08.744776 2341 raft_consensus.cc:491] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.745334 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 890) took 1037 ms. Trace:
I20250115 14:52:08.745400 1751 rpcz_store.cc:276] 0115 14:52:07.708273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.708381 (+ 108us) service_pool.cc:225] Handling call
0115 14:52:08.745248 (+1036867us) ts_tablet_manager.cc:1145] Deleting tablet 8f8efcf1a2db44469cd8dda0b59c34ba
0115 14:52:08.745256 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.745320 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1036693}
I20250115 14:52:08.746433 1751 tablet_replica.cc:331] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.746605 1751 raft_consensus.cc:2238] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.746735 1751 raft_consensus.cc:2267] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.747167 1751 ts_tablet_manager.cc:1905] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.696458 893 tablet_service.cc:1467] Processing CreateTablet for tablet 221b3271c2734828a96c4c86ab031dfc (DEFAULT_TABLE table=default.Table_ea36c03e003e440d99efe3c4c33838fe [id=11553e13f9914ddab5608ec6e2752db7]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:08.750674 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ba21cf54bc074d95b456fd70a9286001 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba21cf54bc074d95b456fd70a9286001 already in progress: creating tablet
W20250115 14:52:08.750957 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 already in progress: creating tablet
W20250115 14:52:08.751209 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 8f8efcf1a2db44469cd8dda0b59c34ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f8efcf1a2db44469cd8dda0b59c34ba already in progress: creating tablet
I20250115 14:52:08.757426 1162 tablet_bootstrap.cc:654] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.757932 1168 ts_tablet_manager.cc:1397] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.003s
I20250115 14:52:08.759284 1751 ts_tablet_manager.cc:1918] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.759462 1751 log.cc:1198] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/83415f1c07ad495cabd287bdcd54dad4
I20250115 14:52:08.760021 1751 ts_tablet_manager.cc:1939] T 83415f1c07ad495cabd287bdcd54dad4 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.760846 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 891) took 1043 ms. Trace:
I20250115 14:52:08.761059 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb866ca7cf2d4d39a46cae8532ed0418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.764068 2241 raft_consensus.cc:491] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.764201 2241 raft_consensus.cc:513] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:08.764869 2241 leader_election.cc:290] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.772130 2241 raft_consensus.cc:491] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.772276 2241 raft_consensus.cc:513] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.772895 2241 leader_election.cc:290] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.780984 2396 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef770dae727b42f5a9513b6f3d68b84b)
I20250115 14:52:08.783057 2396 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } }
I20250115 14:52:08.765632 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a000ab73dd46ac8e134784a156040a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.769482 1168 raft_consensus.cc:357] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.786687 1168 raft_consensus.cc:383] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.786765 1168 raft_consensus.cc:738] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.786963 1168 consensus_queue.cc:260] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.787499 1168 ts_tablet_manager.cc:1428] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20250115 14:52:08.787792 1168 tablet_bootstrap.cc:492] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.773870 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2d04209fd34918b83978325a5e1df8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.770507 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570a78d2b40246ac8f460dcca569ec9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.790284 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221b3271c2734828a96c4c86ab031dfc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:08.792876 983 leader_election.cc:343] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:08.793110 983 leader_election.cc:343] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.795068 1168 tablet_bootstrap.cc:654] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.764917 2341 raft_consensus.cc:513] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.798682 748 tablet_service.cc:1467] Processing CreateTablet for tablet 221b3271c2734828a96c4c86ab031dfc (DEFAULT_TABLE table=default.Table_ea36c03e003e440d99efe3c4c33838fe [id=11553e13f9914ddab5608ec6e2752db7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.799121 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221b3271c2734828a96c4c86ab031dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.800348 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2d510cf7315e46bab6863551ac6bf816"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.800544 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b1ed6d26296a4f9799945452887c3c1c" } modify_external_catalogs: true
I20250115 14:52:08.800753 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b1ed6d26296a4f9799945452887c3c1c" } modify_external_catalogs: true
I20250115 14:52:08.801469 741 tablet_service.cc:1467] Processing CreateTablet for tablet 2087b4ce466f4167b7c32c7074dcf6e7 (DEFAULT_TABLE table=default.Table_32080fc25c67476db569c3357f008e2c [id=9b453c31d5cd4e6a805d95bd66faa8b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.801415 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b8c3f1232b1a40d0a1110de0eb3289a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.801816 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2087b4ce466f4167b7c32c7074dcf6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.802078 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3d9f1f67737c458aa1d0004f77e93808" } modify_external_catalogs: true
I20250115 14:52:08.802235 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3d9f1f67737c458aa1d0004f77e93808" } modify_external_catalogs: true
I20250115 14:52:08.804056 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_132293624dd74f678fbc0f2204143d28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.804368 751 tablet_service.cc:1467] Processing CreateTablet for tablet 98137e1c6ffa4352af8abca6f8d5748d (DEFAULT_TABLE table=default.Table_f6993d1e422647d59233c375c8894af3 [id=aee53ca6780f4992996e152bf26ba4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.804533 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7bc6f9ea70464e1cb86a77b5fb2f48dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.804747 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98137e1c6ffa4352af8abca6f8d5748d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.804917 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1a17daf7f1be4dcd938b8a2e2f7156d1" } new_table_name: "default.Table_2db1454b07e445bd8ecb6db90e9ab6a9" modify_external_catalogs: true
I20250115 14:52:08.805202 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ead9eb0fd5884f6abacc69ebb3213e77"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.809690 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6203efb862ea4962a6d09f3e4306a92f
W20250115 14:52:08.810694 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d510cf7315e46bab6863551ac6bf816 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:08.811856 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8c3f1232b1a40d0a1110de0eb3289a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.811690 450 catalog_manager.cc:5526] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:08.812204 450 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet be2fd31f74a94787a5e8ccae3c5ef245 (table default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec [id=724c015c8f3f4aa581ac6faa84187b06]). Expected version 1 got 0
I20250115 14:52:08.813369 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_303f3f6394c54dd2b5a622b59dee47cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:08.813870 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_303f3f6394c54dd2b5a622b59dee47cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:08.814937 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ead9eb0fd5884f6abacc69ebb3213e77 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:08.815784 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7bc6f9ea70464e1cb86a77b5fb2f48dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:08.816627 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_132293624dd74f678fbc0f2204143d28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.817600 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 6203efb862ea4962a6d09f3e4306a92f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.819293 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2d04209fd34918b83978325a5e1df8" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.819607 2322 consensus_queue.cc:1035] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:08.819751 2341 leader_election.cc:290] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:08.820514 847 leader_election.cc:343] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 9e95d09efd2e44cc81cdcbf87ee453fe
I20250115 14:52:08.820958 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a000ab73dd46ac8e134784a156040a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.821419 984 leader_election.cc:343] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.821521 984 leader_election.cc:304] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.821961 2273 raft_consensus.cc:2743] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.823244 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 6203efb862ea4962a6d09f3e4306a92f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.827464 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e95d09efd2e44cc81cdcbf87ee453fe" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.760991 1751 rpcz_store.cc:276] 0115 14:52:07.717005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.717167 (+ 162us) service_pool.cc:225] Handling call
0115 14:52:08.746394 (+1029227us) ts_tablet_manager.cc:1145] Deleting tablet 83415f1c07ad495cabd287bdcd54dad4
0115 14:52:08.746401 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.753602 (+ 7201us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.759184 (+ 5582us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.760825 (+ 1641us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029051}
I20250115 14:52:08.828754 1751 tablet_replica.cc:331] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.828900 2241 raft_consensus.cc:491] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.829046 2241 raft_consensus.cc:513] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.829656 2241 leader_election.cc:290] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.830385 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5177265ba974ad6a381adb1e8c49589" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.830693 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 847) took 1433 ms. Trace:
I20250115 14:52:08.830777 1167 rpcz_store.cc:276] 0115 14:52:07.397078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.397120 (+ 42us) service_pool.cc:225] Handling call
0115 14:52:08.796427 (+1399307us) ts_tablet_manager.cc:1145] Deleting tablet 064f890830304758a6840a1502e59080
0115 14:52:08.829047 (+ 32620us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.830676 (+ 1629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1399182}
W20250115 14:52:08.830998 983 leader_election.cc:343] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.831346 1751 raft_consensus.cc:2238] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.831527 1751 raft_consensus.cc:2267] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.832742 1751 ts_tablet_manager.cc:1905] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:08.834067 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 064f890830304758a6840a1502e59080 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 064f890830304758a6840a1502e59080 already in progress: creating tablet
I20250115 14:52:08.834960 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 83415f1c07ad495cabd287bdcd54dad4 (table default.Table_458a573b4a4b4714950723b426f0d265 [id=1fd54f7dafdc4777bd06982391452889]) successfully deleted
W20250115 14:52:08.835400 848 leader_election.cc:343] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.835512 848 leader_election.cc:304] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.836176 2341 raft_consensus.cc:2743] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.837262 1167 tablet_replica.cc:331] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:08.837483 1167 raft_consensus.cc:2238] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.837674 1167 raft_consensus.cc:2267] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.840631 1167 ts_tablet_manager.cc:1905] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.843176 1168 tablet_bootstrap.cc:492] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:08.843351 1168 ts_tablet_manager.cc:1397] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.001s
I20250115 14:52:08.844105 1168 raft_consensus.cc:357] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.844394 1168 raft_consensus.cc:383] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.844480 1168 raft_consensus.cc:738] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.844758 1168 consensus_queue.cc:260] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.845266 1168 ts_tablet_manager.cc:1428] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:08.847328 1168 tablet_bootstrap.cc:492] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.853878 1751 ts_tablet_manager.cc:1918] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:08.855475 2365 log.cc:926] Time spent T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
W20250115 14:52:08.857465 984 leader_election.cc:343] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.858685 984 leader_election.cc:304] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.859280 2241 raft_consensus.cc:2743] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.860206 1751 log.cc:1198] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/377d711f052545ddbe499afe7ba3de28
I20250115 14:52:08.857142 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5177265ba974ad6a381adb1e8c49589" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:08.861456 984 leader_election.cc:343] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.861441 1435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1087 } } ignore_live_leader: false dest_uuid: "1cc2e23c560f4096be7b14006bdfc525" is_pre_election: true
I20250115 14:52:08.861573 984 leader_election.cc:304] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.862553 2241 raft_consensus.cc:2743] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.863036 2241 raft_consensus.cc:491] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.863159 2241 raft_consensus.cc:513] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.863751 2241 leader_election.cc:290] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.863988 1751 ts_tablet_manager.cc:1939] T 377d711f052545ddbe499afe7ba3de28 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.864744 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca128bf3d6fc430298da932a78f54184" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:08.865300 983 leader_election.cc:343] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.866667 1166 ts_tablet_manager.cc:1918] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.866693 2396 leader_election.cc:290] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef770dae727b42f5a9513b6f3d68b84b (127.0.104.62:35583), 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227)
I20250115 14:52:08.869684 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 892) took 1152 ms. Trace:
I20250115 14:52:08.870150 1751 rpcz_store.cc:276] 0115 14:52:07.717467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.717510 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:08.828680 (+1111170us) ts_tablet_manager.cc:1145] Deleting tablet 377d711f052545ddbe499afe7ba3de28
0115 14:52:08.828688 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.850860 (+ 22172us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.853790 (+ 2930us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.869570 (+ 15780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1111025}
I20250115 14:52:08.871690 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 377d711f052545ddbe499afe7ba3de28 (table default.Table_7ec0e7f2e4de4b14bd6cea8489658118 [id=898cecdad4254997a5bed59d46f7acec]) successfully deleted
I20250115 14:52:08.872680 1751 tablet_replica.cc:331] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.874678 1751 raft_consensus.cc:2238] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.875301 1751 raft_consensus.cc:2267] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.877532 1751 ts_tablet_manager.cc:1905] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.878332 2341 raft_consensus.cc:491] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.878456 2341 raft_consensus.cc:513] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.879040 2341 leader_election.cc:290] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:08.880219 847 leader_election.cc:343] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e5a059e2f0654a4aafe9e8082b8fde7a
I20250115 14:52:08.877743 1162 tablet_bootstrap.cc:492] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.881393 1162 ts_tablet_manager.cc:1397] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.157s user 0.002s sys 0.009s
I20250115 14:52:08.886281 1162 raft_consensus.cc:357] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.887509 1162 raft_consensus.cc:383] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.888127 1162 raft_consensus.cc:738] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.889427 1162 consensus_queue.cc:260] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.897704 1168 tablet_bootstrap.cc:654] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.866369 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca128bf3d6fc430298da932a78f54184" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.900799 1166 log.cc:1198] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/3a0d97fa57484b9d8cb879fedda61612
I20250115 14:52:08.901202 1166 ts_tablet_manager.cc:1939] T 3a0d97fa57484b9d8cb879fedda61612 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:08.902308 1167 ts_tablet_manager.cc:1918] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.903375 1167 log.cc:1198] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/efb8b001dd66456d9e49e9e8a6848ccf
I20250115 14:52:08.903812 1167 ts_tablet_manager.cc:1939] T efb8b001dd66456d9e49e9e8a6848ccf P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:08.877755 1164 tablet_bootstrap.cc:492] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:08.904697 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 849) took 1504 ms. Trace:
I20250115 14:52:08.904876 1164 ts_tablet_manager.cc:1397] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.405s user 0.003s sys 0.003s
I20250115 14:52:08.905138 1167 rpcz_store.cc:276] 0115 14:52:07.400246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.400317 (+ 71us) service_pool.cc:225] Handling call
0115 14:52:08.832001 (+1431684us) ts_tablet_manager.cc:1145] Deleting tablet efb8b001dd66456d9e49e9e8a6848ccf
0115 14:52:08.837221 (+ 5220us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.898725 (+ 61504us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.902208 (+ 3483us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.904678 (+ 2470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1431554}
I20250115 14:52:08.906538 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet efb8b001dd66456d9e49e9e8a6848ccf (table default.Table_e8f8acbb30db4f019ebeadda783b434a [id=877cb5d7df394006822f9a0e8a133dcc]) successfully deleted
I20250115 14:52:08.907110 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 850) took 1506 ms. Trace:
I20250115 14:52:08.907512 1167 rpcz_store.cc:276] 0115 14:52:07.400701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.401065 (+ 364us) service_pool.cc:225] Handling call
0115 14:52:08.906980 (+1505915us) ts_tablet_manager.cc:1145] Deleting tablet 78a2d39d0296468db48b3eec374244ca
0115 14:52:08.906992 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.907091 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1505781}
W20250115 14:52:08.908243 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 78a2d39d0296468db48b3eec374244ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78a2d39d0296468db48b3eec374244ca already in progress: creating tablet
I20250115 14:52:08.908800 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 851) took 1507 ms. Trace:
I20250115 14:52:08.908973 1164 raft_consensus.cc:357] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.909194 1167 rpcz_store.cc:276] 0115 14:52:07.401402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.401439 (+ 37us) service_pool.cc:225] Handling call
0115 14:52:08.908667 (+1507228us) ts_tablet_manager.cc:1145] Deleting tablet f5c1f0e0b6ee4448b9ba2f78d0117197
0115 14:52:08.908676 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.908784 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1507113}
I20250115 14:52:08.909198 2325 tablet.cc:1721] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:52:08.910233 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f5c1f0e0b6ee4448b9ba2f78d0117197 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5c1f0e0b6ee4448b9ba2f78d0117197 already in progress: creating tablet
I20250115 14:52:08.910604 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 852) took 1508 ms. Trace:
I20250115 14:52:08.910671 1167 rpcz_store.cc:276] 0115 14:52:07.401797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.401835 (+ 38us) service_pool.cc:225] Handling call
0115 14:52:08.910500 (+1508665us) ts_tablet_manager.cc:1145] Deleting tablet 45f124134eab4babac145404c62e7d2a
0115 14:52:08.910509 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.910587 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1508551}
W20250115 14:52:08.911069 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 45f124134eab4babac145404c62e7d2a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 45f124134eab4babac145404c62e7d2a already in progress: creating tablet
I20250115 14:52:08.911737 1167 tablet_replica.cc:331] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:08.911921 1167 raft_consensus.cc:2238] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.912050 1167 raft_consensus.cc:2267] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.913734 465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1087 } } ignore_live_leader: false dest_uuid: "ef770dae727b42f5a9513b6f3d68b84b" is_pre_election: true
I20250115 14:52:08.893467 1162 ts_tablet_manager.cc:1428] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20250115 14:52:08.920634 1167 ts_tablet_manager.cc:1905] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.920930 1162 tablet_bootstrap.cc:492] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.923149 1162 tablet_bootstrap.cc:654] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.924535 1892 leader_election.cc:304] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d97cf5f1fbb4bc6883711020e2fa3f3; no voters: 1cc2e23c560f4096be7b14006bdfc525, ef770dae727b42f5a9513b6f3d68b84b
I20250115 14:52:08.927198 1164 raft_consensus.cc:383] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.928251 1164 raft_consensus.cc:738] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.928906 1164 consensus_queue.cc:260] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.928566 2241 raft_consensus.cc:491] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.929143 2241 raft_consensus.cc:513] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.929984 2241 leader_election.cc:290] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.930629 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eee3d3dfec464baf10c238babb9eb1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:08.931212 983 leader_election.cc:343] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.932669 1164 ts_tablet_manager.cc:1428] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250115 14:52:08.935621 1168 tablet_bootstrap.cc:492] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:08.935832 1168 ts_tablet_manager.cc:1397] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.007s
I20250115 14:52:08.936164 1162 tablet_bootstrap.cc:492] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:08.936370 1162 ts_tablet_manager.cc:1397] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20250115 14:52:08.936622 1164 tablet_bootstrap.cc:492] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:08.937172 1162 raft_consensus.cc:357] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.937366 1162 raft_consensus.cc:383] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:52:08.927528 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 75392.9us
I20250115 14:52:08.937424 1162 raft_consensus.cc:738] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.916255 1751 ts_tablet_manager.cc:1918] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:08.937600 1751 log.cc:1198] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6942d102069d4c33b23fa2c0b29f94a6
I20250115 14:52:08.937929 1751 ts_tablet_manager.cc:1939] T 6942d102069d4c33b23fa2c0b29f94a6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:08.938319 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ff771f5acdcf4fcb9e099aede920f0a1
I20250115 14:52:08.940239 2326 tablet.cc:1721] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:08.940412 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 3a0d97fa57484b9d8cb879fedda61612 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]) successfully deleted
I20250115 14:52:08.940551 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d18a88c8f3ce4a24b7e1e13966760364
W20250115 14:52:08.940657 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 84844b5f50af408ab7dda73f7421cd9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84844b5f50af408ab7dda73f7421cd9d already in progress: creating tablet
W20250115 14:52:08.940884 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 32cb8768ca424aea87682295f30830a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32cb8768ca424aea87682295f30830a9 already in progress: creating tablet
W20250115 14:52:08.941207 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 251b953705c844dda3628d5d5464d4a1 consensus state has no leader
I20250115 14:52:08.941263 2403 raft_consensus.cc:491] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.941413 880 tablet_service.cc:1514] Processing DeleteTablet for tablet d18a88c8f3ce4a24b7e1e13966760364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.941383 2403 raft_consensus.cc:513] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.942001 2403 leader_election.cc:290] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.942446 752 tablet_service.cc:1467] Processing CreateTablet for tablet 570a78d2b40246ac8f460dcca569ec9d (DEFAULT_TABLE table=default.Table_9e22bc322f6e4b8bbcbdb9e756b33180 [id=6378f2ec20c745b7a55607adc9ec3b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.942826 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 570a78d2b40246ac8f460dcca569ec9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.943198 448 catalog_manager.cc:5526] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d7ecece2ab49e1be7079216517d13b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d2d7ecece2ab49e1be7079216517d13b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:08.943598 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:08.942739 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 897) took 1179 ms. Trace:
I20250115 14:52:08.942574 1162 consensus_queue.cc:260] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.945927 1751 rpcz_store.cc:276] 0115 14:52:07.763155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.763291 (+ 136us) service_pool.cc:225] Handling call
0115 14:52:08.872522 (+1109231us) ts_tablet_manager.cc:1145] Deleting tablet 6942d102069d4c33b23fa2c0b29f94a6
0115 14:52:08.872535 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.913299 (+ 40764us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.916159 (+ 2860us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:08.942712 (+ 26553us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1108954}
I20250115 14:52:08.946688 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6942d102069d4c33b23fa2c0b29f94a6 (table default.Table_25a5a732ec814f24893d09a6f0b38d2c [id=fa886a7e8fcd43a096892c6435e13c29]) successfully deleted
I20250115 14:52:08.947753 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e9bb748dd0854fe6ab5fb3dad43da484" } new_table_name: "default.Table_def81b30f40f4877a7e329da62ee3b27" modify_external_catalogs: true
I20250115 14:52:08.948839 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3ac29fd7d7614bd98b6ab37861ad016e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:08.948958 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet d18a88c8f3ce4a24b7e1e13966760364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:08.949366 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ac29fd7d7614bd98b6ab37861ad016e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:08.950361 2241 raft_consensus.cc:491] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.950480 2241 raft_consensus.cc:513] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.951013 2308 catalog_manager.cc:1260] Loaded cluster ID: 838a9b5bdaf340e2b93e4cbb7e53a2a4
I20250115 14:52:08.951062 2241 leader_election.cc:290] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.951108 2308 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: loading cluster ID for follower catalog manager: success
I20250115 14:52:08.951853 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742792be08444792892bd9cd7336dda3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:08.952039 740 tablet_service.cc:1467] Processing CreateTablet for tablet 3f99d417ef074f978a91236d31552315 (DEFAULT_TABLE table=default.Table_044eb9404b5446298ecd6e8f48215b59 [id=22ec9e3bab0b4e07b65749816b82dc48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.952445 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f99d417ef074f978a91236d31552315. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:08.952441 983 leader_election.cc:343] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.952864 748 tablet_service.cc:1467] Processing CreateTablet for tablet 26d7f86ae5e547ecbfece0cb23d7c656 (DEFAULT_TABLE table=default.Table_e01109831b5f43a7b10352cdb82cc440 [id=5bf10dfd595f48ee9bfee83a865fb314]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.953667 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26d7f86ae5e547ecbfece0cb23d7c656. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.954092 1162 ts_tablet_manager.cc:1428] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250115 14:52:08.954375 1162 tablet_bootstrap.cc:492] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:08.954747 893 tablet_service.cc:1514] Processing DeleteTablet for tablet ff771f5acdcf4fcb9e099aede920f0a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:08.955020 1751 tablet_replica.cc:331] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:08.955200 1751 raft_consensus.cc:2238] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:08.955441 1751 raft_consensus.cc:2267] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:08.955972 1751 ts_tablet_manager.cc:1905] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.957355 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet ff771f5acdcf4fcb9e099aede920f0a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:08.958458 1168 raft_consensus.cc:357] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.958765 1168 raft_consensus.cc:383] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:08.958868 1168 raft_consensus.cc:738] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:08.960142 1168 consensus_queue.cc:260] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.960783 1168 ts_tablet_manager.cc:1428] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250115 14:52:08.962143 749 tablet_service.cc:1467] Processing CreateTablet for tablet cb866ca7cf2d4d39a46cae8532ed0418 (DEFAULT_TABLE table=default.Table_cb164ba6d8f74120ae308a71c1a936db [id=43cbf44ddd7f41b7909fcab2320002ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:08.962518 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb866ca7cf2d4d39a46cae8532ed0418. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:08.965623 1168 tablet_bootstrap.cc:492] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:08.967242 1168 tablet_bootstrap.cc:654] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.968606 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 6203efb862ea4962a6d09f3e4306a92f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.970373 1164 tablet_bootstrap.cc:654] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:08.969568 1843 consensus_queue.cc:1035] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:52:08.978369 984 leader_election.cc:343] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:08.978511 984 leader_election.cc:304] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:08.978937 2241 raft_consensus.cc:2743] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:08.986340 2407 tablet.cc:1721] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:08.989328 2326 tablet.cc:1721] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:08.992462 2396 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250115 14:52:08.996420 2241 raft_consensus.cc:491] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:08.996562 2241 raft_consensus.cc:513] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:08.997146 2241 leader_election.cc:290] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:08.999382 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1ce59e05054a009cd36e12db35d449" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:08.999956 983 leader_election.cc:343] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.011430 1167 ts_tablet_manager.cc:1918] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.011647 1167 log.cc:1198] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9853dcdfee5f43de84798cfa0bef674e
I20250115 14:52:09.012044 1167 ts_tablet_manager.cc:1939] T 9853dcdfee5f43de84798cfa0bef674e P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:09.013520 1162 tablet_bootstrap.cc:654] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.013935 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c2aefc9f5c604090bb4bd4f2326991ba" } modify_external_catalogs: true
I20250115 14:52:09.014150 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c2aefc9f5c604090bb4bd4f2326991ba" } modify_external_catalogs: true
I20250115 14:52:09.016019 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0610ab6883db4caa82a59fbf030bfedf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.016604 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0610ab6883db4caa82a59fbf030bfedf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.017975 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbd25ad975704fbcae339bf950fd1052" } new_table_name: "default.Table_82da67a1a37346f8bf107ab64ddc755f" modify_external_catalogs: true
I20250115 14:52:08.971451 741 tablet_service.cc:1514] Processing DeleteTablet for tablet d18a88c8f3ce4a24b7e1e13966760364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:08.971848 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a059e2f0654a4aafe9e8082b8fde7a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.012832 2138 consensus_queue.cc:1035] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:08.979672 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eee3d3dfec464baf10c238babb9eb1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:08.980362 772 raft_consensus.cc:3054] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:09.019332 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c71aa4f14fe849e9be7101f2660dcfee" } new_table_name: "default.Table_1ba99d5a8ef746c3baa4c0cd075226e6" modify_external_catalogs: true
I20250115 14:52:09.019564 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 853) took 1617 ms. Trace:
I20250115 14:52:09.019642 1167 rpcz_store.cc:276] 0115 14:52:07.402184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.402220 (+ 36us) service_pool.cc:225] Handling call
0115 14:52:08.911685 (+1509465us) ts_tablet_manager.cc:1145] Deleting tablet 9853dcdfee5f43de84798cfa0bef674e
0115 14:52:08.911693 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:08.926910 (+ 15217us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.011308 (+ 84398us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.019548 (+ 8240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1509349}
I20250115 14:52:09.020098 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9853dcdfee5f43de84798cfa0bef674e (table default.Table_4ec686b46a094818bd820c4f3854220b [id=5dc3e53ee73a4441960dc231d02d163f]) successfully deleted
I20250115 14:52:09.020412 1167 tablet_replica.cc:331] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:09.020629 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ab72fac2354c4767b21a84c1a2fc5dec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:09.020597 1167 raft_consensus.cc:2238] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.020933 1167 raft_consensus.cc:2267] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:09.021119 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab72fac2354c4767b21a84c1a2fc5dec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.021445 1167 ts_tablet_manager.cc:1905] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:08.980650 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742792be08444792892bd9cd7336dda3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.002070 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1ce59e05054a009cd36e12db35d449" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.022356 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b84df010f86642d4b3585c0e774575bf" } new_table_name: "default.Table_8a376696a65648979c5f5791bc936fab" modify_external_catalogs: true
I20250115 14:52:09.006629 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ff771f5acdcf4fcb9e099aede920f0a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.023967 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d6b3a467e1e2479b9734b57db305df67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.024461 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6b3a467e1e2479b9734b57db305df67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.025642 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 994fe2bae335497f8c07f50b91377074 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:09.025938 2138 consensus_queue.cc:1035] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:52:09.026050 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87026b6faf75437eac21dedab1f41ed4 consensus state has no leader
I20250115 14:52:09.026381 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec" } modify_external_catalogs: true
I20250115 14:52:09.026540 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec" } modify_external_catalogs: true
I20250115 14:52:09.028743 2407 tablet.cc:1721] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:52:09.030431 848 consensus_peers.cc:487] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:09.036141 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e842e1c2b44754b3b9a2e026486a84" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:09.036401 911 raft_consensus.cc:2388] T 67e842e1c2b44754b3b9a2e026486a84 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d7ecece2ab49e1be7079216517d13b in current term 1: Already voted for candidate c238c037a1c74725a001e2b4a1ff4304 in this term.
I20250115 14:52:09.036779 2241 raft_consensus.cc:491] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.036917 2241 raft_consensus.cc:513] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.037567 2241 leader_election.cc:290] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.039283 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e842e1c2b44754b3b9a2e026486a84" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:52:09.040138 848 leader_election.cc:343] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.040249 848 leader_election.cc:304] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.041270 2341 raft_consensus.cc:2743] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.042215 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c34919648447fac14208ede0ddc01" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:09.042804 983 leader_election.cc:343] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:09.045444 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6203efb862ea4962a6d09f3e4306a92f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6203efb862ea4962a6d09f3e4306a92f already in progress: creating tablet
W20250115 14:52:09.045709 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d18a88c8f3ce4a24b7e1e13966760364 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d18a88c8f3ce4a24b7e1e13966760364 already in progress: creating tablet
W20250115 14:52:09.045913 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ff771f5acdcf4fcb9e099aede920f0a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff771f5acdcf4fcb9e099aede920f0a1 already in progress: creating tablet
I20250115 14:52:09.067162 1168 tablet_bootstrap.cc:492] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:09.067270 1164 tablet_bootstrap.cc:492] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:09.067385 1168 ts_tablet_manager.cc:1397] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.002s
I20250115 14:52:09.068166 1164 ts_tablet_manager.cc:1397] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.000s
I20250115 14:52:09.068390 1168 raft_consensus.cc:357] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.068673 1168 raft_consensus.cc:383] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.068809 1168 raft_consensus.cc:738] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.069062 1168 consensus_queue.cc:260] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.069859 1168 ts_tablet_manager.cc:1428] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:09.070158 1168 tablet_bootstrap.cc:492] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:09.070348 710 leader_election.cc:304] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:09.070946 2404 raft_consensus.cc:2743] T 67e842e1c2b44754b3b9a2e026486a84 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.071434 1162 tablet_bootstrap.cc:492] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.071607 1162 ts_tablet_manager.cc:1397] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.117s user 0.002s sys 0.002s
I20250115 14:52:09.072206 1162 raft_consensus.cc:357] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.073743 1162 raft_consensus.cc:383] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.073860 1162 raft_consensus.cc:738] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.074081 1162 consensus_queue.cc:260] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.074653 1162 ts_tablet_manager.cc:1428] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:52:09.074940 1162 tablet_bootstrap.cc:492] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.075037 1751 ts_tablet_manager.cc:1918] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.075124 1751 log.cc:1198] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/95718d39a7824517a36c49c3b86fef30
I20250115 14:52:09.075381 1751 ts_tablet_manager.cc:1939] T 95718d39a7824517a36c49c3b86fef30 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.076053 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 898) took 1312 ms. Trace:
I20250115 14:52:09.077354 1164 raft_consensus.cc:357] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.077543 1164 raft_consensus.cc:383] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.077606 1164 raft_consensus.cc:738] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.079641 2325 tablet.cc:1721] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:52:09.080312 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a005ad11e7941fab96b5565d55cfaf7 consensus state has no leader
I20250115 14:52:09.083701 453 master_service.cc:946] ReplaceTablet: received request to replace tablet e7315a41489a4f8296ad8b12c039b0a2 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:09.084722 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 3f1e5d44b8064abbbf1b6419a1eca1ed from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:09.086011 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f2ba4086a6bf4bfa83e740b31dfb53f8
I20250115 14:52:09.086562 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 0ee6906a03f546bbaaecea44cf0e7a68 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:09.087180 886 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ba4086a6bf4bfa83e740b31dfb53f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.092025 1164 consensus_queue.cc:260] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.093513 752 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ba4086a6bf4bfa83e740b31dfb53f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:09.095389 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f2ba4086a6bf4bfa83e740b31dfb53f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2ba4086a6bf4bfa83e740b31dfb53f8 already in progress: creating tablet
I20250115 14:52:09.094933 1164 ts_tablet_manager.cc:1428] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250115 14:52:09.076138 1751 rpcz_store.cc:276] 0115 14:52:07.763249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.763497 (+ 248us) service_pool.cc:225] Handling call
0115 14:52:08.954973 (+1191476us) ts_tablet_manager.cc:1145] Deleting tablet 95718d39a7824517a36c49c3b86fef30
0115 14:52:08.954984 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.051318 (+ 96334us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.074926 (+ 23608us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.076036 (+ 1110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191353}
I20250115 14:52:09.096724 1164 tablet_bootstrap.cc:492] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:09.086862 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ba4086a6bf4bfa83e740b31dfb53f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.097059 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 95718d39a7824517a36c49c3b86fef30 (table default.Table_5d8548b30284461086fdcd351454ad57 [id=d114031263ea4c87a6c4d12c9d6a0d32]) successfully deleted
I20250115 14:52:09.097560 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 900) took 1269 ms. Trace:
I20250115 14:52:09.097656 1751 rpcz_store.cc:276] 0115 14:52:07.828237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.835550 (+ 7313us) service_pool.cc:225] Handling call
0115 14:52:09.097406 (+1261856us) ts_tablet_manager.cc:1145] Deleting tablet 325cd3fd8b4d447998822163e575e758
0115 14:52:09.097418 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.097534 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1261597}
W20250115 14:52:09.098116 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 325cd3fd8b4d447998822163e575e758 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 325cd3fd8b4d447998822163e575e758 already in progress: creating tablet
I20250115 14:52:09.099321 1167 ts_tablet_manager.cc:1918] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.099439 1167 log.cc:1198] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/4abde784f9c948cb96e6cc93c30817bd
W20250115 14:52:09.099599 984 leader_election.cc:343] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.099711 1167 ts_tablet_manager.cc:1939] T 4abde784f9c948cb96e6cc93c30817bd P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:09.099797 984 leader_election.cc:304] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.100024 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 854) took 1697 ms. Trace:
I20250115 14:52:09.100111 1167 rpcz_store.cc:276] 0115 14:52:07.402531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.402567 (+ 36us) service_pool.cc:225] Handling call
0115 14:52:09.020364 (+1617797us) ts_tablet_manager.cc:1145] Deleting tablet 4abde784f9c948cb96e6cc93c30817bd
0115 14:52:09.020374 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.077284 (+ 56910us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.099210 (+ 21926us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.100005 (+ 795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1617679}
I20250115 14:52:09.100200 2418 raft_consensus.cc:2743] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.100493 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 4abde784f9c948cb96e6cc93c30817bd (table default.Table_10deaf92a667422fa029a731f755dfef [id=e657e05ca35045f1a196dd012ee59483]) successfully deleted
W20250115 14:52:09.100795 984 leader_election.cc:343] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.101493 1168 tablet_bootstrap.cc:654] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.101609 984 leader_election.cc:304] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.101871 2273 raft_consensus.cc:2743] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:09.102275 984 leader_election.cc:343] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.102779 984 leader_election.cc:304] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.103056 2273 raft_consensus.cc:2743] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.102351 1162 tablet_bootstrap.cc:654] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.100991 2418 consensus_queue.cc:1035] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:09.104938 1167 tablet_replica.cc:331] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:09.105178 1167 raft_consensus.cc:2238] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.105374 1167 raft_consensus.cc:2267] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.105918 1167 ts_tablet_manager.cc:1905] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.101116 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 901) took 1268 ms. Trace:
I20250115 14:52:09.109155 1751 rpcz_store.cc:276] 0115 14:52:07.832483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.835842 (+ 3359us) service_pool.cc:225] Handling call
0115 14:52:09.100999 (+1265157us) ts_tablet_manager.cc:1145] Deleting tablet c042a4f4f9bc46afb0b94df8ed74a8cf
0115 14:52:09.101010 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.101098 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1264967}
W20250115 14:52:09.109694 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c042a4f4f9bc46afb0b94df8ed74a8cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c042a4f4f9bc46afb0b94df8ed74a8cf already in progress: creating tablet
I20250115 14:52:09.109773 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 911) took 1092 ms. Trace:
I20250115 14:52:09.109848 1751 rpcz_store.cc:276] 0115 14:52:08.017549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.020122 (+ 2573us) service_pool.cc:225] Handling call
0115 14:52:09.109629 (+1089507us) ts_tablet_manager.cc:1145] Deleting tablet 005a4cd0e9f14e76b01afa49d946ee32
0115 14:52:09.109642 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.109754 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089323}
W20250115 14:52:09.110196 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 005a4cd0e9f14e76b01afa49d946ee32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 005a4cd0e9f14e76b01afa49d946ee32 already in progress: creating tablet
I20250115 14:52:09.110275 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 910) took 1092 ms. Trace:
I20250115 14:52:09.110337 1751 rpcz_store.cc:276] 0115 14:52:08.017398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.023675 (+ 6277us) service_pool.cc:225] Handling call
0115 14:52:09.110175 (+1086500us) ts_tablet_manager.cc:1145] Deleting tablet 2f7a5cbd9a78485a816d7fecdc23106a
0115 14:52:09.110184 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.110261 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1086311}
I20250115 14:52:09.110674 1751 tablet_replica.cc:331] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:09.110656 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2f7a5cbd9a78485a816d7fecdc23106a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f7a5cbd9a78485a816d7fecdc23106a already in progress: creating tablet
I20250115 14:52:09.110863 1751 raft_consensus.cc:2238] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.111012 1751 raft_consensus.cc:2267] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.111243 1843 raft_consensus.cc:491] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.111367 1843 raft_consensus.cc:513] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.111559 1751 ts_tablet_manager.cc:1905] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.111956 1843 leader_election.cc:290] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:09.115396 2418 raft_consensus.cc:491] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.115556 2418 raft_consensus.cc:513] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.116696 2418 leader_election.cc:290] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.112146 2340 raft_consensus.cc:491] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.117105 2340 raft_consensus.cc:513] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.117663 2340 leader_election.cc:290] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:09.113168 847 leader_election.cc:343] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: fa57c1fa96594a40b2551ad503127e4f
I20250115 14:52:09.116560 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc363f70f3f44d309f69fac9356c6257" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:09.118939 983 leader_election.cc:343] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:09.119443 847 leader_election.cc:343] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 95718d39a7824517a36c49c3b86fef30
I20250115 14:52:09.124109 2326 tablet.cc:1721] T fbf94077e0e343b8938b081102dfd2af P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:09.125105 2407 tablet.cc:1721] T fbf94077e0e343b8938b081102dfd2af P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:09.126564 1751 ts_tablet_manager.cc:1918] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.128688 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa57c1fa96594a40b2551ad503127e4f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.130085 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95718d39a7824517a36c49c3b86fef30" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.130430 1162 tablet_bootstrap.cc:492] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.132421 1167 ts_tablet_manager.cc:1918] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.132453 1162 ts_tablet_manager.cc:1397] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.000s
I20250115 14:52:09.132613 1167 log.cc:1198] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/759a4ebe7b0444128ddd31a78aad463e
I20250115 14:52:09.133169 1162 raft_consensus.cc:357] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.133368 1162 raft_consensus.cc:383] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.133420 1162 raft_consensus.cc:738] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.133626 1162 consensus_queue.cc:260] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.134140 1167 ts_tablet_manager.cc:1939] T 759a4ebe7b0444128ddd31a78aad463e P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:09.134151 1162 ts_tablet_manager.cc:1428] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:09.134408 1162 tablet_bootstrap.cc:492] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.135358 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 855) took 1732 ms. Trace:
I20250115 14:52:09.136159 1162 tablet_bootstrap.cc:654] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.136839 1751 log.cc:1198] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/07a000ab73dd46ac8e134784a156040a
I20250115 14:52:09.137215 1751 ts_tablet_manager.cc:1939] T 07a000ab73dd46ac8e134784a156040a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.137581 1167 rpcz_store.cc:276] 0115 14:52:07.402868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.402905 (+ 37us) service_pool.cc:225] Handling call
0115 14:52:09.104885 (+1701980us) ts_tablet_manager.cc:1145] Deleting tablet 759a4ebe7b0444128ddd31a78aad463e
0115 14:52:09.104896 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.121511 (+ 16615us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.124249 (+ 2738us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.135340 (+ 11091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1701854}
I20250115 14:52:09.138340 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 759a4ebe7b0444128ddd31a78aad463e (table default.Table_d1345e860a054b708e11f2f64347a3d7 [id=87e2283ba18a4525ab55a8316238aac7]) successfully deleted
I20250115 14:52:09.143302 1167 tablet_replica.cc:331] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:09.143577 1167 raft_consensus.cc:2238] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:09.145076 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet cee072daff2d4943b68261806666e4a0 (table default.Table_59bae91b591d4af2879199b9fbf81bf5 [id=e64b65f9e24e4223a8dd6f688478ec87], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet cee072daff2d4943b68261806666e4a0 consensus state has no leader
I20250115 14:52:09.145112 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 905) took 1167 ms. Trace:
I20250115 14:52:09.145198 1751 rpcz_store.cc:276] 0115 14:52:07.977505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.992825 (+ 15320us) service_pool.cc:225] Handling call
0115 14:52:09.110629 (+1117804us) ts_tablet_manager.cc:1145] Deleting tablet 07a000ab73dd46ac8e134784a156040a
0115 14:52:09.110637 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.119381 (+ 8744us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.126472 (+ 7091us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.145088 (+ 18616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1058521}
I20250115 14:52:09.149515 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 07a000ab73dd46ac8e134784a156040a (table default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]) successfully deleted
I20250115 14:52:09.149960 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 912) took 1074 ms. Trace:
I20250115 14:52:09.150044 1751 rpcz_store.cc:276] 0115 14:52:08.075356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.075410 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:09.149816 (+1074406us) ts_tablet_manager.cc:1145] Deleting tablet e910c387e6a24caf95e14b57c8b83e6f
0115 14:52:09.149826 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.149941 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1070550}
W20250115 14:52:09.150496 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e910c387e6a24caf95e14b57c8b83e6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e910c387e6a24caf95e14b57c8b83e6f already in progress: creating tablet
I20250115 14:52:09.150858 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 913) took 1073 ms. Trace:
I20250115 14:52:09.150929 1751 rpcz_store.cc:276] 0115 14:52:08.077119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.077174 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:09.150752 (+1073578us) ts_tablet_manager.cc:1145] Deleting tablet b54807a4e9644b86a89670f57ae154a3
0115 14:52:09.150761 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.150842 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1071404}
W20250115 14:52:09.151326 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b54807a4e9644b86a89670f57ae154a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b54807a4e9644b86a89670f57ae154a3 already in progress: creating tablet
W20250115 14:52:09.151877 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d360add9d39a48caa7b90f7d786f7bc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d360add9d39a48caa7b90f7d786f7bc1 already in progress: creating tablet
I20250115 14:52:09.152220 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_58c17e0ad22e41738519c725c25dcdfc" } modify_external_catalogs: true
I20250115 14:52:09.152437 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_58c17e0ad22e41738519c725c25dcdfc" } modify_external_catalogs: true
I20250115 14:52:09.153877 2273 consensus_queue.cc:1035] T 67e842e1c2b44754b3b9a2e026486a84 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:52:09.154843 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 210f161cf6b94a2abbad651c40bb8cec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 210f161cf6b94a2abbad651c40bb8cec already in progress: creating tablet
W20250115 14:52:09.155568 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e5dfc00ecec54f92a75fa77c84fe8a6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5dfc00ecec54f92a75fa77c84fe8a6d already in progress: creating tablet
W20250115 14:52:09.156401 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 929830d8e8fd4b0c8a7b18c941706929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 929830d8e8fd4b0c8a7b18c941706929 already in progress: creating tablet
W20250115 14:52:09.156518 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1e3879802464acaa279dfe4fbb634bb consensus state has no leader
I20250115 14:52:09.156802 447 master_service.cc:946] ReplaceTablet: received request to replace tablet b3c31e24081d4a42aa933acbb80c6495 from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:09.157359 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a349cfaff6be4b85a0b2cf3eee238c26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a349cfaff6be4b85a0b2cf3eee238c26 already in progress: creating tablet
I20250115 14:52:09.157440 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_062a9fcca27a4d90b73742167b635ae8" } modify_external_catalogs: true
I20250115 14:52:09.157589 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_062a9fcca27a4d90b73742167b635ae8" } modify_external_catalogs: true
W20250115 14:52:09.158185 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 1140016d11a74a2cadba494c0b319c54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1140016d11a74a2cadba494c0b319c54 already in progress: creating tablet
I20250115 14:52:09.158953 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet be2fd31f74a94787a5e8ccae3c5ef245
I20250115 14:52:09.159543 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae] alter complete (version 1)
I20250115 14:52:09.160503 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e7315a41489a4f8296ad8b12c039b0a2
I20250115 14:52:09.160933 453 catalog_manager.cc:6596] ReplaceTablet: tablet e7315a41489a4f8296ad8b12c039b0a2 deleted and replaced by tablet 0b252e48108947cbbb36335499b51c92
I20250115 14:52:09.161330 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3f1e5d44b8064abbbf1b6419a1eca1ed
I20250115 14:52:09.161643 446 catalog_manager.cc:6596] ReplaceTablet: tablet 3f1e5d44b8064abbbf1b6419a1eca1ed deleted and replaced by tablet 5b33c876fab1476aba3b1f6e799fa96c
I20250115 14:52:09.163640 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet be2fd31f74a94787a5e8ccae3c5ef245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.163692 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet e7315a41489a4f8296ad8b12c039b0a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b252e48108947cbbb36335499b51c92) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.163751 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet ba596cfb3ce64718a42a9083975edc8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.163793 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 4294c49da1e044be993100b3cf5e2cea with delete_type TABLET_DATA_DELETED (replaced by tablet d1259256b7744e739581125611abbe67) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.163836 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 076278b53996407c8a5c73a389459eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.163973 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed with delete_type TABLET_DATA_DELETED (replaced by tablet 5b33c876fab1476aba3b1f6e799fa96c) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.158650 1751 tablet_replica.cc:331] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.170086 1751 raft_consensus.cc:2238] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.170522 1751 raft_consensus.cc:2267] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.171274 1751 ts_tablet_manager.cc:1905] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.172745 2418 raft_consensus.cc:491] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.172947 2418 raft_consensus.cc:513] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.174240 2418 leader_election.cc:290] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:09.175215 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 63416.8us
I20250115 14:52:09.175695 1162 tablet_bootstrap.cc:492] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.175817 1162 ts_tablet_manager.cc:1397] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20250115 14:52:09.176191 752 tablet_service.cc:1514] Processing DeleteTablet for tablet e7315a41489a4f8296ad8b12c039b0a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b252e48108947cbbb36335499b51c92) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.176450 1162 raft_consensus.cc:357] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.176610 1162 raft_consensus.cc:383] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.176663 1162 raft_consensus.cc:738] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.176826 740 tablet_service.cc:1514] Processing DeleteTablet for tablet be2fd31f74a94787a5e8ccae3c5ef245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.176837 1162 consensus_queue.cc:260] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.177508 1162 ts_tablet_manager.cc:1428] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:09.177748 1162 tablet_bootstrap.cc:492] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.179076 1162 tablet_bootstrap.cc:654] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.183136 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_449c8d18596548be9715ed0c45d1bc4f" } new_table_name: "default.Table_555a230c62e04d098ec432545d739dba" modify_external_catalogs: true
I20250115 14:52:09.184218 1751 ts_tablet_manager.cc:1918] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.185295 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc363f70f3f44d309f69fac9356c6257" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.186208 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed with delete_type TABLET_DATA_DELETED (replaced by tablet 5b33c876fab1476aba3b1f6e799fa96c) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.186456 1166 tablet_replica.cc:331] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:09.186630 1166 raft_consensus.cc:2238] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.186764 1166 raft_consensus.cc:2267] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.187268 1166 ts_tablet_manager.cc:1905] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.187670 2418 raft_consensus.cc:491] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.187886 2418 raft_consensus.cc:513] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.188787 2418 leader_election.cc:290] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.188755 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f61f66337491436e90983173399d81ae" } modify_external_catalogs: true
I20250115 14:52:09.188956 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f61f66337491436e90983173399d81ae" } modify_external_catalogs: true
I20250115 14:52:09.188867 1751 log.cc:1198] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/1b2d04209fd34918b83978325a5e1df8
I20250115 14:52:09.189311 1751 ts_tablet_manager.cc:1939] T 1b2d04209fd34918b83978325a5e1df8 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.190274 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 1b2d04209fd34918b83978325a5e1df8 (table default.Table_d548be616a2c4f01ac52eadd5c009eaf [id=1907295f0b5c49b9b99a188d4dc9f7f8]) successfully deleted
I20250115 14:52:09.191041 1162 tablet_bootstrap.cc:492] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.191217 1162 ts_tablet_manager.cc:1397] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20250115 14:52:09.191907 1162 raft_consensus.cc:357] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.192200 1162 raft_consensus.cc:383] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.162858 886 tablet_service.cc:1514] Processing DeleteTablet for tablet be2fd31f74a94787a5e8ccae3c5ef245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.174088 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4294c49da1e044be993100b3cf5e2cea" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:09.192463 1162 raft_consensus.cc:738] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.192653 1162 consensus_queue.cc:260] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.192864 1167 raft_consensus.cc:2267] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.193490 1167 ts_tablet_manager.cc:1905] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.198839 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet ea8ab6a34caa486ab929af2cac3ce957 (table default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d]). Expected version 1 got 0
I20250115 14:52:09.201742 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 30f3a87bf8b6452d8afd8dfd836ee4e6
I20250115 14:52:09.202394 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d6f2bc62621d402cacfcbe2eb74173f9
I20250115 14:52:09.202723 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.203332 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f2bc62621d402cacfcbe2eb74173f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.203560 1162 ts_tablet_manager.cc:1428] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250115 14:52:09.203831 1162 tablet_bootstrap.cc:492] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:09.203955 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e7315a41489a4f8296ad8b12c039b0a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7315a41489a4f8296ad8b12c039b0a2 already in progress: creating tablet
W20250115 14:52:09.204738 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet be2fd31f74a94787a5e8ccae3c5ef245 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be2fd31f74a94787a5e8ccae3c5ef245 already in progress: creating tablet
I20250115 14:52:09.205406 1162 tablet_bootstrap.cc:654] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.206099 2308 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: acquiring CA information for follower catalog manager: success
I20250115 14:52:09.209231 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.214969 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c34919648447fac14208ede0ddc01" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.217239 1751 tablet_replica.cc:331] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.217748 1751 raft_consensus.cc:2238] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.217967 1751 raft_consensus.cc:2267] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.218587 1751 ts_tablet_manager.cc:1905] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.218643 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4294c49da1e044be993100b3cf5e2cea" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.219216 740 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f2bc62621d402cacfcbe2eb74173f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.220427 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba596cfb3ce64718a42a9083975edc8d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:09.217494 984 leader_election.cc:343] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.224366 984 leader_election.cc:304] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.224607 2418 raft_consensus.cc:2743] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:09.229686 984 leader_election.cc:343] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:09.230023 984 leader_election.cc:343] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.230202 984 leader_election.cc:304] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.230454 2418 raft_consensus.cc:2743] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:09.230849 984 leader_election.cc:343] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.231642 1166 ts_tablet_manager.cc:1918] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:09.231812 1166 log.cc:1198] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/3f1e5d44b8064abbbf1b6419a1eca1ed
I20250115 14:52:09.232242 1166 ts_tablet_manager.cc:1939] T 3f1e5d44b8064abbbf1b6419a1eca1ed P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:52:09.236112 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99449d50c2dd46159a69dc0af1de7a87 consensus state has no leader
I20250115 14:52:09.236766 880 tablet_service.cc:1514] Processing DeleteTablet for tablet fa57c1fa96594a40b2551ad503127e4f with delete_type TABLET_DATA_DELETED (replaced by tablet 9c35882d2e5c464c9cfad496ba9d6af9) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.236799 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e185963130bf49a1af88ab9e14e8e1bc
I20250115 14:52:09.237370 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c1602aef514dd29d5d2cffb043dbb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:09.238606 983 leader_election.cc:343] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.238730 983 leader_election.cc:304] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.239622 2418 raft_consensus.cc:2743] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.240243 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164] alter complete (version 1)
I20250115 14:52:09.240928 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet e185963130bf49a1af88ab9e14e8e1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.242218 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 95718d39a7824517a36c49c3b86fef30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.244354 1166 tablet_replica.cc:331] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:09.244566 1166 raft_consensus.cc:2238] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:09.245384 1166 raft_consensus.cc:2267] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.245993 1166 ts_tablet_manager.cc:1905] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.249176 1164 tablet_bootstrap.cc:654] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.238349 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba596cfb3ce64718a42a9083975edc8d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:09.251806 1162 tablet_bootstrap.cc:492] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.251992 1162 ts_tablet_manager.cc:1397] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.000s
I20250115 14:52:09.252281 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a059e2f0654a4aafe9e8082b8fde7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.252527 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed with delete_type TABLET_DATA_DELETED (replaced by tablet 5b33c876fab1476aba3b1f6e799fa96c) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.252719 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.252873 886 tablet_service.cc:1514] Processing DeleteTablet for tablet e7315a41489a4f8296ad8b12c039b0a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 0b252e48108947cbbb36335499b51c92) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:09.253142 983 leader_election.cc:343] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.253263 983 leader_election.cc:304] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.253400 1162 raft_consensus.cc:357] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.253646 2418 raft_consensus.cc:2743] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.253650 1162 raft_consensus.cc:383] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:52:09.253950 848 leader_election.cc:343] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.254014 1162 raft_consensus.cc:738] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.254063 848 leader_election.cc:304] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.254261 1162 consensus_queue.cc:260] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.254590 2340 raft_consensus.cc:2743] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.254809 882 tablet_service.cc:1514] Processing DeleteTablet for tablet e185963130bf49a1af88ab9e14e8e1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.255923 886 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f2bc62621d402cacfcbe2eb74173f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:09.257266 848 leader_election.cc:343] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.257368 848 leader_election.cc:304] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.257753 1843 raft_consensus.cc:2743] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.258491 1162 ts_tablet_manager.cc:1428] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250115 14:52:09.258783 1162 tablet_bootstrap.cc:492] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.260353 1162 tablet_bootstrap.cc:654] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.262049 740 tablet_service.cc:1514] Processing DeleteTablet for tablet e185963130bf49a1af88ab9e14e8e1bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:09.262938 848 consensus_peers.cc:487] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f1e5d44b8064abbbf1b6419a1eca1ed. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:09.264686 2325 tablet.cc:1721] T fbf94077e0e343b8938b081102dfd2af P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:09.266841 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 3f1e5d44b8064abbbf1b6419a1eca1ed (table default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 [id=cdf45f410fda46f29cd6dbfd1cc45f28]) successfully deleted
I20250115 14:52:09.271628 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_64bd6866f7624f7eaecc8e0486d14b21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.272337 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64bd6866f7624f7eaecc8e0486d14b21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.273078 1751 ts_tablet_manager.cc:1918] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.273202 1751 log.cc:1198] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b5177265ba974ad6a381adb1e8c49589
I20250115 14:52:09.273499 1751 ts_tablet_manager.cc:1939] T b5177265ba974ad6a381adb1e8c49589 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.274497 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b5177265ba974ad6a381adb1e8c49589 (table default.Table_b6a520bf69264c44b1da30c0ffb41cc0 [id=f0b642abaec54506b98190fd1ac9ce67]) successfully deleted
W20250115 14:52:09.274612 848 consensus_peers.cc:487] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43605c995df4400e91139199c9514bfd. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:09.277448 1751 tablet_replica.cc:331] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.277683 1751 raft_consensus.cc:2238] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.277879 1751 raft_consensus.cc:2267] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.278390 1751 ts_tablet_manager.cc:1905] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.279940 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8bab10f63d3c43eca6cda9f4c2d4189a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.280541 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bab10f63d3c43eca6cda9f4c2d4189a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.281458 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_482c3b93feb244538f5483b3419a7a60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.281919 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_482c3b93feb244538f5483b3419a7a60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.282756 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0be69f26c9784abb83eee94fe5bfdc39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.283227 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0be69f26c9784abb83eee94fe5bfdc39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.283629 1166 ts_tablet_manager.cc:1918] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:09.283749 1166 log.cc:1198] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/30f3a87bf8b6452d8afd8dfd836ee4e6
I20250115 14:52:09.284061 1166 ts_tablet_manager.cc:1939] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:09.284253 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_af7ebd20849b40729c5650194d55240f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.284709 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af7ebd20849b40729c5650194d55240f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.285657 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ae6d939a2d5e4be482722ce8098cb5d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.286180 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae6d939a2d5e4be482722ce8098cb5d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.288612 1168 tablet_bootstrap.cc:492] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:09.289682 1164 tablet_bootstrap.cc:492] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:09.289896 1164 ts_tablet_manager.cc:1397] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.193s user 0.002s sys 0.002s
I20250115 14:52:09.289875 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_82da67a1a37346f8bf107ab64ddc755f [id=00a89ba1b93f4132aae8fe6e748df77e] alter complete (version 3)
I20250115 14:52:09.292268 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet ea8ab6a34caa486ab929af2cac3ce957 (table default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d]). Expected version 1 got 0
I20250115 14:52:09.294289 1164 raft_consensus.cc:357] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.294567 1164 raft_consensus.cc:383] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.294835 1164 raft_consensus.cc:738] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.295114 1164 consensus_queue.cc:260] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.296154 1164 ts_tablet_manager.cc:1428] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:52:09.291060 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet b803d8e103a5423f85c75c9006809b17 with delete_type TABLET_DATA_DELETED (replaced by tablet 6437873733584722979c1e9ae8b0730b) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.291201 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce498ffd3f04c5d8eb642739a838903 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.296537 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ea8ab6a34caa486ab929af2cac3ce957 (table default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d]). Expected version 1 got 0
I20250115 14:52:09.298214 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3991253a712b4579981713c7f964a053"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.298755 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3991253a712b4579981713c7f964a053 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.300235 1168 ts_tablet_manager.cc:1397] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.230s user 0.015s sys 0.026s
I20250115 14:52:09.301020 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 759a4ebe7b0444128ddd31a78aad463e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.298216 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 83415f1c07ad495cabd287bdcd54dad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.293862 1167 ts_tablet_manager.cc:1918] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.301463 1167 log.cc:1198] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/50273a0019bb4cd8977f7ef014372d85
I20250115 14:52:09.301829 1167 ts_tablet_manager.cc:1939] T 50273a0019bb4cd8977f7ef014372d85 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:09.303386 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 856) took 1900 ms. Trace:
I20250115 14:52:09.314625 1167 rpcz_store.cc:276] 0115 14:52:07.403224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.403261 (+ 37us) service_pool.cc:225] Handling call
0115 14:52:09.143240 (+1739979us) ts_tablet_manager.cc:1145] Deleting tablet 50273a0019bb4cd8977f7ef014372d85
0115 14:52:09.143253 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.196858 (+ 53605us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.292692 (+ 95834us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.303366 (+ 10674us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1739861}
I20250115 14:52:09.317396 1751 ts_tablet_manager.cc:1918] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.317451 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 50273a0019bb4cd8977f7ef014372d85 (table default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]) successfully deleted
I20250115 14:52:09.317523 1751 log.cc:1198] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4f1ce59e05054a009cd36e12db35d449
I20250115 14:52:09.314291 1168 raft_consensus.cc:357] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.317777 1751 ts_tablet_manager.cc:1939] T 4f1ce59e05054a009cd36e12db35d449 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.317893 1168 raft_consensus.cc:383] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.318004 1168 raft_consensus.cc:738] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.318262 1168 consensus_queue.cc:260] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.318624 1167 tablet_replica.cc:331] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:09.318931 1167 raft_consensus.cc:2238] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.321415 1167 raft_consensus.cc:2267] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.321960 1167 ts_tablet_manager.cc:1905] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.322150 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 4f1ce59e05054a009cd36e12db35d449 (table default.Table_e209cc3424eb4349b5387a32ac96d6bc [id=18a209f910d44ef081b5925cd6ebe028]) successfully deleted
I20250115 14:52:09.322675 1751 tablet_replica.cc:331] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.322865 1751 raft_consensus.cc:2238] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.321254 1168 ts_tablet_manager.cc:1428] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250115 14:52:09.323000 1751 raft_consensus.cc:2267] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.323045 2340 raft_consensus.cc:491] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.323179 2340 raft_consensus.cc:513] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.323277 1168 tablet_bootstrap.cc:492] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:09.323451 1751 ts_tablet_manager.cc:1905] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.323712 2340 leader_election.cc:290] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:09.325405 847 leader_election.cc:343] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 4341977f363d40eeac71b7d33d1b43ac
I20250115 14:52:09.329020 2273 raft_consensus.cc:491] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.329169 2273 raft_consensus.cc:513] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.329538 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4341977f363d40eeac71b7d33d1b43ac" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.331537 1168 tablet_bootstrap.cc:654] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.332612 2418 raft_consensus.cc:491] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.333191 1164 tablet_bootstrap.cc:492] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:09.334800 1164 tablet_bootstrap.cc:654] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.338915 2273 leader_election.cc:290] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.339102 2418 raft_consensus.cc:513] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.339242 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9b57b8b9434f55ace5a4f47380df54" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.339666 2418 leader_election.cc:290] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:09.332015 848 leader_election.cc:343] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.340005 848 leader_election.cc:304] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.339951 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9b57b8b9434f55ace5a4f47380df54" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:09.340246 1166 tablet_replica.cc:331] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:09.340343 2340 raft_consensus.cc:2743] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.340459 1166 raft_consensus.cc:2238] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.340591 1166 raft_consensus.cc:2267] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:09.341130 983 leader_election.cc:343] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.341351 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3f8cae6244968927b864e7ce97565" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:09.341748 983 leader_election.cc:343] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.342043 1166 ts_tablet_manager.cc:1905] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:09.342229 984 leader_election.cc:343] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.342321 984 leader_election.cc:304] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.342648 2418 raft_consensus.cc:2743] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.343109 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3f8cae6244968927b864e7ce97565" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:09.343576 984 leader_election.cc:343] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.343662 984 leader_election.cc:304] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.343971 2418 raft_consensus.cc:2743] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.346312 2404 raft_consensus.cc:491] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.346431 2404 raft_consensus.cc:513] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.346971 2404 leader_election.cc:290] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.347988 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ce65604c294021af93632ed073f627" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:09.348165 911 raft_consensus.cc:2463] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:09.348765 710 leader_election.cc:304] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
W20250115 14:52:09.349018 710 leader_election.cc:343] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d9ce65604c294021af93632ed073f627
I20250115 14:52:09.349368 2404 raft_consensus.cc:2798] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:09.349438 2404 raft_consensus.cc:491] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:09.349489 2404 raft_consensus.cc:3054] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:09.350669 1167 ts_tablet_manager.cc:1918] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.350757 1167 log.cc:1198] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/01291d41a2b345b4bfd00e6be0fb0129
I20250115 14:52:09.351009 1167 ts_tablet_manager.cc:1939] T 01291d41a2b345b4bfd00e6be0fb0129 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:09.351713 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 857) took 1948 ms. Trace:
I20250115 14:52:09.351789 1167 rpcz_store.cc:276] 0115 14:52:07.403556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.403594 (+ 38us) service_pool.cc:225] Handling call
0115 14:52:09.318571 (+1914977us) ts_tablet_manager.cc:1145] Deleting tablet 01291d41a2b345b4bfd00e6be0fb0129
0115 14:52:09.318582 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.342657 (+ 24075us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.350576 (+ 7919us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.351695 (+ 1119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1914853}
I20250115 14:52:09.351855 2404 raft_consensus.cc:513] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.352244 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 01291d41a2b345b4bfd00e6be0fb0129 (table default.Table_21fa1a9206d64c8bb19a405f37518a65 [id=e31104604e8648c08fd706e44150cda1]) successfully deleted
I20250115 14:52:09.352892 2404 leader_election.cc:290] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.353436 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ce65604c294021af93632ed073f627" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:09.353602 911 raft_consensus.cc:3054] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:09.356087 911 raft_consensus.cc:2463] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:09.356145 1167 tablet_replica.cc:331] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:09.356333 1167 raft_consensus.cc:2238] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.356887 1167 raft_consensus.cc:2267] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.356621 710 leader_election.cc:304] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
W20250115 14:52:09.357542 710 leader_election.cc:343] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: d9ce65604c294021af93632ed073f627
I20250115 14:52:09.357582 1167 ts_tablet_manager.cc:1905] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.357846 2403 raft_consensus.cc:2798] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:09.357954 2403 raft_consensus.cc:695] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:09.358181 2403 consensus_queue.cc:237] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.360421 1751 ts_tablet_manager.cc:1918] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.360527 1751 log.cc:1198] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/c37f4c2ab9d248a1af817cfa2c2e2ff0
I20250115 14:52:09.360790 1751 ts_tablet_manager.cc:1939] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.361692 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 (table default.Table_6f0b6c724c1c43e7adf538ad7369b7a1 [id=9e8ce51e784e4bdbb3ac78cba10bfedf]) successfully deleted
I20250115 14:52:09.368688 1168 tablet_bootstrap.cc:492] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:09.372438 2308 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250115 14:52:09.373498 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7654d2831f0141858b52379b767b3a70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7654d2831f0141858b52379b767b3a70 already in progress: creating tablet
W20250115 14:52:09.374261 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 221b3271c2734828a96c4c86ab031dfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 221b3271c2734828a96c4c86ab031dfc already in progress: creating tablet
I20250115 14:52:09.377808 1168 ts_tablet_manager.cc:1397] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.000s
I20250115 14:52:09.378727 1168 raft_consensus.cc:357] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.379292 1168 raft_consensus.cc:383] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.379611 1168 raft_consensus.cc:738] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.380234 1168 consensus_queue.cc:260] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.380813 2340 raft_consensus.cc:491] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.382275 2340 raft_consensus.cc:513] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.382912 2340 leader_election.cc:290] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:09.383443 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83415f1c07ad495cabd287bdcd54dad4" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:09.384019 848 leader_election.cc:343] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.388131 1168 ts_tablet_manager.cc:1428] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250115 14:52:09.388484 1168 tablet_bootstrap.cc:492] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:09.389470 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f400880835ab4efdb2af0be5553d20dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f400880835ab4efdb2af0be5553d20dc already in progress: creating tablet
W20250115 14:52:09.390194 847 leader_election.cc:343] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 83415f1c07ad495cabd287bdcd54dad4
I20250115 14:52:09.390308 847 leader_election.cc:304] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.390734 2340 raft_consensus.cc:2743] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.391037 1751 tablet_replica.cc:331] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.391227 1751 raft_consensus.cc:2238] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.391362 1751 raft_consensus.cc:2267] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.391852 1751 ts_tablet_manager.cc:1905] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.394819 1162 tablet_bootstrap.cc:492] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.394961 1162 ts_tablet_manager.cc:1397] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.136s user 0.004s sys 0.011s
I20250115 14:52:09.396610 1164 tablet_bootstrap.cc:492] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:09.396807 1164 ts_tablet_manager.cc:1397] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.010s
I20250115 14:52:09.400640 1162 raft_consensus.cc:357] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.400861 1162 raft_consensus.cc:383] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.400885 1168 tablet_bootstrap.cc:654] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.400918 1162 raft_consensus.cc:738] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.408442 1162 consensus_queue.cc:260] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.410785 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8e26a69364458794f8848ffe16a71f (DEFAULT_TABLE table=default.Table_cc65d5ae73344c87b1bcf475ebac6b00 [id=1305addcf5434611926744efb365f453]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.411201 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8e26a69364458794f8848ffe16a71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.412878 740 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8e26a69364458794f8848ffe16a71f (DEFAULT_TABLE table=default.Table_cc65d5ae73344c87b1bcf475ebac6b00 [id=1305addcf5434611926744efb365f453]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.413241 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8e26a69364458794f8848ffe16a71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.416538 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 374da44997ca441f91b0c0bceeded415 (DEFAULT_TABLE table=default.Table_fd29d5224a264121b8da1f6186711fd7 [id=a27ac34fa7d2483b98d4e9f27e98afd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.416675 1162 ts_tablet_manager.cc:1428] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20250115 14:52:09.416848 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 374da44997ca441f91b0c0bceeded415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.419840 1167 ts_tablet_manager.cc:1918] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.419941 1167 log.cc:1198] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d9ce65604c294021af93632ed073f627
I20250115 14:52:09.420526 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 (DEFAULT_TABLE table=default.Table_18573a0a9c86499d9223ac71aa866a24 [id=76742c377d7e43599c196c9caedf45c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.420830 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.422391 740 tablet_service.cc:1467] Processing CreateTablet for tablet 374da44997ca441f91b0c0bceeded415 (DEFAULT_TABLE table=default.Table_fd29d5224a264121b8da1f6186711fd7 [id=a27ac34fa7d2483b98d4e9f27e98afd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.422714 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 374da44997ca441f91b0c0bceeded415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.425451 893 tablet_service.cc:1467] Processing CreateTablet for tablet 63b149ce2d7a45339cc67f28d1cb4224 (DEFAULT_TABLE table=default.Table_fa193ea0c83741f9af1c5110e55af30a [id=b45e86a8cd9f43bb819128b22b0151e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.425608 882 tablet_service.cc:1467] Processing CreateTablet for tablet 374da44997ca441f91b0c0bceeded415 (DEFAULT_TABLE table=default.Table_fd29d5224a264121b8da1f6186711fd7 [id=a27ac34fa7d2483b98d4e9f27e98afd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.425807 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63b149ce2d7a45339cc67f28d1cb4224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.426049 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 374da44997ca441f91b0c0bceeded415. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.425446 880 tablet_service.cc:1467] Processing CreateTablet for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 (DEFAULT_TABLE table=default.Table_18573a0a9c86499d9223ac71aa866a24 [id=76742c377d7e43599c196c9caedf45c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.427250 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.427255 1162 tablet_bootstrap.cc:492] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:09.428627 848 consensus_peers.cc:487] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:09.428795 1162 tablet_bootstrap.cc:654] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.425451 886 tablet_service.cc:1467] Processing CreateTablet for tablet 2e8e26a69364458794f8848ffe16a71f (DEFAULT_TABLE table=default.Table_cc65d5ae73344c87b1bcf475ebac6b00 [id=1305addcf5434611926744efb365f453]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.432077 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8e26a69364458794f8848ffe16a71f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.434206 752 tablet_service.cc:1467] Processing CreateTablet for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 (DEFAULT_TABLE table=default.Table_18573a0a9c86499d9223ac71aa866a24 [id=76742c377d7e43599c196c9caedf45c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.434670 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.440881 752 tablet_service.cc:1467] Processing CreateTablet for tablet 63b149ce2d7a45339cc67f28d1cb4224 (DEFAULT_TABLE table=default.Table_fa193ea0c83741f9af1c5110e55af30a [id=b45e86a8cd9f43bb819128b22b0151e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.441289 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63b149ce2d7a45339cc67f28d1cb4224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.443956 1751 ts_tablet_manager.cc:1918] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.444075 1751 log.cc:1198] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/742792be08444792892bd9cd7336dda3
I20250115 14:52:09.444371 1751 ts_tablet_manager.cc:1939] T 742792be08444792892bd9cd7336dda3 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.445392 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 742792be08444792892bd9cd7336dda3 (table default.Table_97c066067f7a49cab2dd5bdade9f6de2 [id=8c1c9324a0cb4ce5b5f64af989e4bdf6]) successfully deleted
I20250115 14:52:09.447252 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 99ae2f8bd8fe4e92995dddf414172f0f (DEFAULT_TABLE table=default.Table_4a8a9e000ade430aaa44bad5dd61df9f [id=602481fd6f244746948b9ff4b61d272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.447613 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ae2f8bd8fe4e92995dddf414172f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.449967 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_062a9fcca27a4d90b73742167b635ae8 [id=5d05be5a21a44fe0b75c284bf3c83625]) successfully deleted
W20250115 14:52:09.450268 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d6f2bc62621d402cacfcbe2eb74173f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6f2bc62621d402cacfcbe2eb74173f9 already in progress: creating tablet
W20250115 14:52:09.451331 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e185963130bf49a1af88ab9e14e8e1bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e185963130bf49a1af88ab9e14e8e1bc already in progress: creating tablet
I20250115 14:52:09.451566 1167 ts_tablet_manager.cc:1939] T d9ce65604c294021af93632ed073f627 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:09.452906 748 tablet_service.cc:1467] Processing CreateTablet for tablet c2683bc3c4c943b1b1a778befdbc06b4 (DEFAULT_TABLE table=default.Table_22a3195f3aa44230a56a246622e2d946 [id=ab1197a60fc34d1381a9bcafb90c45bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.453406 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 858) took 2049 ms. Trace:
I20250115 14:52:09.453553 1167 rpcz_store.cc:276] 0115 14:52:07.403931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.403984 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:09.356096 (+1952112us) ts_tablet_manager.cc:1145] Deleting tablet d9ce65604c294021af93632ed073f627
0115 14:52:09.356107 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.389001 (+ 32894us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.419747 (+ 30746us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.453388 (+ 33641us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1951993}
I20250115 14:52:09.454072 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d9ce65604c294021af93632ed073f627 (table default.Table_79fecf866fd6405a9b7df222877fc2b0 [id=0748f5a86b034a96b1dfe1cd723d2438]) successfully deleted
I20250115 14:52:09.454461 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 866) took 1852 ms. Trace:
I20250115 14:52:09.455323 741 tablet_service.cc:1467] Processing CreateTablet for tablet 99ae2f8bd8fe4e92995dddf414172f0f (DEFAULT_TABLE table=default.Table_4a8a9e000ade430aaa44bad5dd61df9f [id=602481fd6f244746948b9ff4b61d272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.460355 2403 consensus_queue.cc:1035] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:09.463703 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet ea8ab6a34caa486ab929af2cac3ce957 (table default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d]). Expected version 1 got 0
I20250115 14:52:09.464509 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b3c31e24081d4a42aa933acbb80c6495
I20250115 14:52:09.464870 447 catalog_manager.cc:6596] ReplaceTablet: tablet b3c31e24081d4a42aa933acbb80c6495 deleted and replaced by tablet 1f354e4204ec47149957eb22f964f08a
I20250115 14:52:09.466277 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ea8ab6a34caa486ab929af2cac3ce957 (table default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d]). Expected version 1 got 0
I20250115 14:52:09.466910 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a6f3f8cae6244968927b864e7ce97565 (table default.Table_5d9f72946dd846b98daa1df0c392cb72 [id=02f9f84c80334c36801ef8e760120ab3]). Expected version 2 got 0
I20250115 14:52:09.467175 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e847214582ef4c838ae73922b0c49ef9" } new_table_name: "default.Table_4fe9c35efdec44e1b24937fe275c2b3c" modify_external_catalogs: true
I20250115 14:52:09.467741 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 994fe2bae335497f8c07f50b91377074
I20250115 14:52:09.468088 451 catalog_manager.cc:6596] ReplaceTablet: tablet 994fe2bae335497f8c07f50b91377074 deleted and replaced by tablet 51c3aa6f1d944ce782cee0507c7b1f6c
I20250115 14:52:09.468093 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1b68293cce4a49c3b939caef23d5e36c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.468837 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b68293cce4a49c3b939caef23d5e36c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.470592 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0ee6906a03f546bbaaecea44cf0e7a68
I20250115 14:52:09.470969 455 catalog_manager.cc:6596] ReplaceTablet: tablet 0ee6906a03f546bbaaecea44cf0e7a68 deleted and replaced by tablet 061234cb76b842a7a374ddd5f44fd9a1
I20250115 14:52:09.471285 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_707a274426cd4d37a8ee193d73a0e171" } new_table_name: "default.Table_c118c528732d4868a96f871f5ab7e2bb" modify_external_catalogs: true
I20250115 14:52:09.472539 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1f9e67f6de614b0ba6cd1403e8c3ef5b" } modify_external_catalogs: true
I20250115 14:52:09.472509 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6f324c73cb6341b68a7b0d3947954ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:09.472644 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1f9e67f6de614b0ba6cd1403e8c3ef5b" } modify_external_catalogs: true
W20250115 14:52:09.472971 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f324c73cb6341b68a7b0d3947954ae0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.473765 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f459a15fa0234ba9b5722b7bbff3b6cb" } new_table_name: "default.Table_f2433c7e11464f4ba59d5895f819e337" modify_external_catalogs: true
I20250115 14:52:09.472200 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_74edc08b682e4c5cb696e54fcadaf5f1" } modify_external_catalogs: true
I20250115 14:52:09.474644 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_74edc08b682e4c5cb696e54fcadaf5f1" } modify_external_catalogs: true
I20250115 14:52:09.477948 885 tablet_service.cc:1467] Processing CreateTablet for tablet d90be0a60c0a4eeca3f59afa9cfa4688 (DEFAULT_TABLE table=default.Table_fd4d720185214c7cbec302eaf573e265 [id=e976991e2fdb4163a57726f3df675713]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.478219 876 tablet_service.cc:1467] Processing CreateTablet for tablet 097644179d224707bb5061e4d6b28fe3 (DEFAULT_TABLE table=default.Table_b29e828cea18463e9e2b28ab06b9f1aa [id=9b683fac2caf46bcb57b222f61480e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.478315 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d90be0a60c0a4eeca3f59afa9cfa4688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.478475 875 tablet_service.cc:1467] Processing CreateTablet for tablet 36ac2c9c80824c25bb974092b3aba744 (DEFAULT_TABLE table=default.Table_3592cd57b6b945288efcb39b7753ab52 [id=df271c4faa7d4e0d957a69578d772458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.478646 883 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c31e24081d4a42aa933acbb80c6495 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f354e4204ec47149957eb22f964f08a) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.478672 879 tablet_service.cc:1467] Processing CreateTablet for tablet 9107b85c08c646cab34f2128ea8db84f (DEFAULT_TABLE table=default.Table_5ce3a92ee6154dee8af40b3ac625550c [id=74a3f9e259514a4ab2ddfb67c1c0714c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.477911 891 tablet_service.cc:1467] Processing CreateTablet for tablet df0d072330444439a6b9b87db099c8b1 (DEFAULT_TABLE table=default.Table_2c0b01c3b5394ebbac4a5e930276ed6c [id=2a505c2b70024b21b8b42727cab3d028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.478839 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee6906a03f546bbaaecea44cf0e7a68 with delete_type TABLET_DATA_DELETED (replaced by tablet 061234cb76b842a7a374ddd5f44fd9a1) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.478492 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 097644179d224707bb5061e4d6b28fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.479156 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 994fe2bae335497f8c07f50b91377074 with delete_type TABLET_DATA_DELETED (replaced by tablet 51c3aa6f1d944ce782cee0507c7b1f6c) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.479166 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9107b85c08c646cab34f2128ea8db84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.479435 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0d072330444439a6b9b87db099c8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.480556 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 097644179d224707bb5061e4d6b28fe3 (DEFAULT_TABLE table=default.Table_b29e828cea18463e9e2b28ab06b9f1aa [id=9b683fac2caf46bcb57b222f61480e6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.480890 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 097644179d224707bb5061e4d6b28fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.482913 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec4ce555ee04d17af66067a7712e19b (DEFAULT_TABLE table=default.Table_2ffb08cfa3fa4eea9582ee724bc15f9b [id=d1d1a5c4824d419cbc85aadbb4192b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.483203 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec4ce555ee04d17af66067a7712e19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.484522 1011 tablet_service.cc:1467] Processing CreateTablet for tablet d90be0a60c0a4eeca3f59afa9cfa4688 (DEFAULT_TABLE table=default.Table_fd4d720185214c7cbec302eaf573e265 [id=e976991e2fdb4163a57726f3df675713]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.484804 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d90be0a60c0a4eeca3f59afa9cfa4688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.486701 1014 tablet_service.cc:1467] Processing CreateTablet for tablet df0d072330444439a6b9b87db099c8b1 (DEFAULT_TABLE table=default.Table_2c0b01c3b5394ebbac4a5e930276ed6c [id=2a505c2b70024b21b8b42727cab3d028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.487022 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0d072330444439a6b9b87db099c8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.488533 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 9107b85c08c646cab34f2128ea8db84f (DEFAULT_TABLE table=default.Table_5ce3a92ee6154dee8af40b3ac625550c [id=74a3f9e259514a4ab2ddfb67c1c0714c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.488844 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9107b85c08c646cab34f2128ea8db84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.492573 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 36ac2c9c80824c25bb974092b3aba744 (DEFAULT_TABLE table=default.Table_3592cd57b6b945288efcb39b7753ab52 [id=df271c4faa7d4e0d957a69578d772458]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.492870 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ac2c9c80824c25bb974092b3aba744. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.493495 1751 tablet_replica.cc:331] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.495461 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_be9185f83999438bae9ef0d3dca88fd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:09.493697 1751 raft_consensus.cc:2238] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:09.496029 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be9185f83999438bae9ef0d3dca88fd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.496145 1751 raft_consensus.cc:2267] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.496711 1751 ts_tablet_manager.cc:1905] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.496826 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ac2c9c80824c25bb974092b3aba744. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.505698 880 tablet_service.cc:1467] Processing CreateTablet for tablet c2683bc3c4c943b1b1a778befdbc06b4 (DEFAULT_TABLE table=default.Table_22a3195f3aa44230a56a246622e2d946 [id=ab1197a60fc34d1381a9bcafb90c45bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.506063 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2683bc3c4c943b1b1a778befdbc06b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.506898 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 994fe2bae335497f8c07f50b91377074 with delete_type TABLET_DATA_DELETED (replaced by tablet 51c3aa6f1d944ce782cee0507c7b1f6c) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.507344 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 580742eb5dac4dba9995809890e0e315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.507512 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b82924e735846b2914520684d1b8fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.507803 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee6906a03f546bbaaecea44cf0e7a68 with delete_type TABLET_DATA_DELETED (replaced by tablet 061234cb76b842a7a374ddd5f44fd9a1) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.508558 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 63b149ce2d7a45339cc67f28d1cb4224 (DEFAULT_TABLE table=default.Table_fa193ea0c83741f9af1c5110e55af30a [id=b45e86a8cd9f43bb819128b22b0151e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.509369 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63b149ce2d7a45339cc67f28d1cb4224. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.509672 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f367ac1a82704324a73f7079165a6e45" } modify_external_catalogs: true
I20250115 14:52:09.509850 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f367ac1a82704324a73f7079165a6e45" } modify_external_catalogs: true
I20250115 14:52:09.512619 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec4ce555ee04d17af66067a7712e19b (DEFAULT_TABLE table=default.Table_2ffb08cfa3fa4eea9582ee724bc15f9b [id=d1d1a5c4824d419cbc85aadbb4192b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.513010 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec4ce555ee04d17af66067a7712e19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.513854 2418 raft_consensus.cc:491] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.513981 2418 raft_consensus.cc:513] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.514559 2418 leader_election.cc:290] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.515332 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0651be54e0b244a4bace9f47efc9b58a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:09.515852 983 leader_election.cc:343] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.519676 882 tablet_service.cc:1467] Processing CreateTablet for tablet 99ae2f8bd8fe4e92995dddf414172f0f (DEFAULT_TABLE table=default.Table_4a8a9e000ade430aaa44bad5dd61df9f [id=602481fd6f244746948b9ff4b61d272b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.520083 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ae2f8bd8fe4e92995dddf414172f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.524502 2340 raft_consensus.cc:491] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.524616 2340 raft_consensus.cc:513] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.525198 2340 leader_election.cc:290] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:09.525806 886 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab00385cc884cb98cabee780619a32f (DEFAULT_TABLE table=default.Table_5a3e279adb9548baa2d7264f270977e3 [id=c637ef8083214e69b33cf7ed6c54dc91]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:09.526113 847 leader_election.cc:343] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 90c1602aef514dd29d5d2cffb043dbb6
I20250115 14:52:09.526762 1028 tablet_service.cc:1467] Processing CreateTablet for tablet c2683bc3c4c943b1b1a778befdbc06b4 (DEFAULT_TABLE table=default.Table_22a3195f3aa44230a56a246622e2d946 [id=ab1197a60fc34d1381a9bcafb90c45bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.527166 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2683bc3c4c943b1b1a778befdbc06b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.527683 1751 ts_tablet_manager.cc:1918] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.527798 1751 log.cc:1198] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ff9b57b8b9434f55ace5a4f47380df54
I20250115 14:52:09.528070 1751 ts_tablet_manager.cc:1939] T ff9b57b8b9434f55ace5a4f47380df54 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.532104 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_60dfab3c6fc946b3afe6a4e168627e2c" } new_table_name: "default.Table_f79a705788cf4992a6b1be7d153f13f6" modify_external_catalogs: true
I20250115 14:52:09.532457 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab00385cc884cb98cabee780619a32f (DEFAULT_TABLE table=default.Table_5a3e279adb9548baa2d7264f270977e3 [id=c637ef8083214e69b33cf7ed6c54dc91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.532868 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab00385cc884cb98cabee780619a32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.536391 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c31e24081d4a42aa933acbb80c6495 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f354e4204ec47149957eb22f964f08a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.526165 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab00385cc884cb98cabee780619a32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.542021 2418 raft_consensus.cc:491] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.542176 2418 raft_consensus.cc:513] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.542806 2418 leader_election.cc:290] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.545480 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b803d8e103a5423f85c75c9006809b17" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:09.546010 983 leader_election.cc:343] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.546120 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 931) took 1075 ms. Trace:
I20250115 14:52:09.546213 1751 rpcz_store.cc:276] 0115 14:52:08.471116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.479296 (+ 8180us) service_pool.cc:225] Handling call
0115 14:52:09.493446 (+1014150us) ts_tablet_manager.cc:1145] Deleting tablet ff9b57b8b9434f55ace5a4f47380df54
0115 14:52:09.493456 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.499548 (+ 6092us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.527599 (+ 28051us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.546101 (+ 18502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1013455}
I20250115 14:52:09.546622 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ff9b57b8b9434f55ace5a4f47380df54 (table default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]) successfully deleted
I20250115 14:52:09.547160 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0651be54e0b244a4bace9f47efc9b58a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.547119 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b803d8e103a5423f85c75c9006809b17" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.547119 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90c1602aef514dd29d5d2cffb043dbb6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:09.547883 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2683bc3c4c943b1b1a778befdbc06b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:09.547915 848 leader_election.cc:343] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.548012 848 leader_election.cc:304] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.548216 2340 raft_consensus.cc:2743] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.456012 1167 rpcz_store.cc:276] 0115 14:52:07.602206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.602255 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:09.454344 (+1852089us) ts_tablet_manager.cc:1145] Deleting tablet e3c54fdc672f4709a44b2542212dfb5e
0115 14:52:09.454355 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.454444 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1845491}
I20250115 14:52:09.551872 1162 tablet_bootstrap.cc:492] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.551999 1162 ts_tablet_manager.cc:1397] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.125s user 0.003s sys 0.000s
W20250115 14:52:09.552033 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:45653, user_credentials={real_user=slave}} blocked reactor thread for 96060.2us
I20250115 14:52:09.552783 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 867) took 1941 ms. Trace:
I20250115 14:52:09.552680 1162 raft_consensus.cc:357] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.552889 1162 raft_consensus.cc:383] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.552937 1162 raft_consensus.cc:738] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.552858 1167 rpcz_store.cc:276] 0115 14:52:07.611058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.619177 (+ 8119us) service_pool.cc:225] Handling call
0115 14:52:09.552658 (+1933481us) ts_tablet_manager.cc:1145] Deleting tablet 96349985df1e49e9b942eb610ccf70f9
0115 14:52:09.552669 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.552764 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1933304}
I20250115 14:52:09.553115 1162 consensus_queue.cc:260] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.553596 1162 ts_tablet_manager.cc:1428] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:09.553858 1162 tablet_bootstrap.cc:492] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.555424 1162 tablet_bootstrap.cc:654] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:09.552783 984 leader_election.cc:343] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.556233 984 leader_election.cc:304] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:09.556638 984 leader_election.cc:343] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.556718 984 leader_election.cc:304] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.557274 2273 raft_consensus.cc:2743] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.557399 2273 raft_consensus.cc:2743] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:09.558990 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e3c54fdc672f4709a44b2542212dfb5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3c54fdc672f4709a44b2542212dfb5e already in progress: creating tablet
I20250115 14:52:09.559404 1751 tablet_replica.cc:331] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.559597 1751 raft_consensus.cc:2238] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:09.559908 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 96349985df1e49e9b942eb610ccf70f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96349985df1e49e9b942eb610ccf70f9 already in progress: creating tablet
I20250115 14:52:09.560621 1751 raft_consensus.cc:2267] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.560844 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 871) took 1901 ms. Trace:
I20250115 14:52:09.561349 1751 ts_tablet_manager.cc:1905] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.562608 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4ec4ce555ee04d17af66067a7712e19b (DEFAULT_TABLE table=default.Table_2ffb08cfa3fa4eea9582ee724bc15f9b [id=d1d1a5c4824d419cbc85aadbb4192b9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.562995 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ec4ce555ee04d17af66067a7712e19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.564574 748 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab00385cc884cb98cabee780619a32f (DEFAULT_TABLE table=default.Table_5a3e279adb9548baa2d7264f270977e3 [id=c637ef8083214e69b33cf7ed6c54dc91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.564896 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ab00385cc884cb98cabee780619a32f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.569893 749 tablet_service.cc:1467] Processing CreateTablet for tablet df0d072330444439a6b9b87db099c8b1 (DEFAULT_TABLE table=default.Table_2c0b01c3b5394ebbac4a5e930276ed6c [id=2a505c2b70024b21b8b42727cab3d028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.570282 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df0d072330444439a6b9b87db099c8b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.573275 1024 tablet_service.cc:1467] Processing CreateTablet for tablet de27f6c600b840c2b00a881c5ef038ae (DEFAULT_TABLE table=default.Table_50430a4659d544eba42d4e0e1cef37da [id=8c66e79985374ed182ce8097914a49d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.573666 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de27f6c600b840c2b00a881c5ef038ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.574764 1164 raft_consensus.cc:357] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.574995 1164 raft_consensus.cc:383] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.575075 1164 raft_consensus.cc:738] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.575295 1164 consensus_queue.cc:260] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.576552 1018 tablet_service.cc:1467] Processing CreateTablet for tablet a9f2e5eadb54419a9071d64474073e34 (DEFAULT_TABLE table=default.Table_f2006bfc8694432482d41dd92edbf093 [id=ee6e2171cc904b0497638242d6b43b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.576936 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f2e5eadb54419a9071d64474073e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.573181 875 tablet_service.cc:1467] Processing CreateTablet for tablet bb859e2a553f4ab3b20452b48e933f0e (DEFAULT_TABLE table=default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.577484 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb859e2a553f4ab3b20452b48e933f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.579833 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 59e5cc832ffa4740aaacd65a565420f5 (DEFAULT_TABLE table=default.Table_3962d1f342824bf990b968aeba430bff [id=709b1a8596274f538f23cf0e9007b857]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.573468 876 tablet_service.cc:1467] Processing CreateTablet for tablet a9f2e5eadb54419a9071d64474073e34 (DEFAULT_TABLE table=default.Table_f2006bfc8694432482d41dd92edbf093 [id=ee6e2171cc904b0497638242d6b43b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.580585 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f2e5eadb54419a9071d64474073e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.580973 1751 ts_tablet_manager.cc:1918] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.581070 1751 log.cc:1198] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/84844b5f50af408ab7dda73f7421cd9d
I20250115 14:52:09.581317 1751 ts_tablet_manager.cc:1939] T 84844b5f50af408ab7dda73f7421cd9d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.581720 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e5cc832ffa4740aaacd65a565420f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.583470 1164 ts_tablet_manager.cc:1428] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.187s user 0.002s sys 0.000s
I20250115 14:52:09.584918 1048 raft_consensus.cc:1270] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:09.585451 2138 consensus_queue.cc:1035] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:09.588702 1167 rpcz_store.cc:276] 0115 14:52:07.659288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.659331 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:09.560716 (+1901385us) ts_tablet_manager.cc:1145] Deleting tablet ba21cf54bc074d95b456fd70a9286001
0115 14:52:09.560727 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.560825 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1901224}
I20250115 14:52:09.589680 749 tablet_service.cc:1467] Processing CreateTablet for tablet 097644179d224707bb5061e4d6b28fe3 (DEFAULT_TABLE table=default.Table_b29e828cea18463e9e2b28ab06b9f1aa [id=9b683fac2caf46bcb57b222f61480e6a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:09.590535 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ba21cf54bc074d95b456fd70a9286001 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba21cf54bc074d95b456fd70a9286001 already in progress: creating tablet
I20250115 14:52:09.590955 1167 tablet_replica.cc:331] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:09.591292 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 097644179d224707bb5061e4d6b28fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.591961 1167 raft_consensus.cc:2238] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.591959 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9107b85c08c646cab34f2128ea8db84f (DEFAULT_TABLE table=default.Table_5ce3a92ee6154dee8af40b3ac625550c [id=74a3f9e259514a4ab2ddfb67c1c0714c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.593077 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9107b85c08c646cab34f2128ea8db84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.593652 1167 raft_consensus.cc:2267] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.593957 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 933) took 1030 ms. Trace:
I20250115 14:52:09.594166 1167 ts_tablet_manager.cc:1905] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.594062 1751 rpcz_store.cc:276] 0115 14:52:08.563049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.571710 (+ 8661us) service_pool.cc:225] Handling call
0115 14:52:09.559344 (+987634us) ts_tablet_manager.cc:1145] Deleting tablet 84844b5f50af408ab7dda73f7421cd9d
0115 14:52:09.559357 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.565751 (+ 6394us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.580883 (+ 15132us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.593928 (+ 13045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":970372}
I20250115 14:52:09.594518 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 84844b5f50af408ab7dda73f7421cd9d (table default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]) successfully deleted
I20250115 14:52:09.595129 1751 tablet_replica.cc:331] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:09.596236 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 32cb8768ca424aea87682295f30830a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32cb8768ca424aea87682295f30830a9 already in progress: creating tablet
I20250115 14:52:09.596392 744 tablet_service.cc:1514] Processing DeleteTablet for tablet b3c31e24081d4a42aa933acbb80c6495 with delete_type TABLET_DATA_DELETED (replaced by tablet 1f354e4204ec47149957eb22f964f08a) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:09.596452 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6203efb862ea4962a6d09f3e4306a92f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6203efb862ea4962a6d09f3e4306a92f already in progress: creating tablet
I20250115 14:52:09.592578 755 tablet_service.cc:1467] Processing CreateTablet for tablet 36ac2c9c80824c25bb974092b3aba744 (DEFAULT_TABLE table=default.Table_3592cd57b6b945288efcb39b7753ab52 [id=df271c4faa7d4e0d957a69578d772458]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:09.596596 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d18a88c8f3ce4a24b7e1e13966760364 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d18a88c8f3ce4a24b7e1e13966760364 already in progress: creating tablet
W20250115 14:52:09.596715 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ff771f5acdcf4fcb9e099aede920f0a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff771f5acdcf4fcb9e099aede920f0a1 already in progress: creating tablet
I20250115 14:52:09.596741 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36ac2c9c80824c25bb974092b3aba744. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:09.596861 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f2ba4086a6bf4bfa83e740b31dfb53f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2ba4086a6bf4bfa83e740b31dfb53f8 already in progress: creating tablet
I20250115 14:52:09.603493 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 994fe2bae335497f8c07f50b91377074 with delete_type TABLET_DATA_DELETED (replaced by tablet 51c3aa6f1d944ce782cee0507c7b1f6c) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.606328 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee6906a03f546bbaaecea44cf0e7a68 with delete_type TABLET_DATA_DELETED (replaced by tablet 061234cb76b842a7a374ddd5f44fd9a1) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.608748 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b85d63974ab418eb1921dd605fbf2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.609148 755 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ce65604c294021af93632ed073f627 with delete_type TABLET_DATA_DELETED (replaced by tablet 339dd408dab7430c980058af3af04433) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:09.612241 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 55200.6us
I20250115 14:52:09.612649 1751 raft_consensus.cc:2238] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.612919 1751 pending_rounds.cc:62] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Trying to abort 2 pending ops.
I20250115 14:52:09.612972 1751 pending_rounds.cc:69] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7114558380373045248 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "be2fd31f74a94787a5e8ccae3c5ef245" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec" dest_uuid: "73c86158cc2641f182ea50a06634592d" new_extra_config { } }
I20250115 14:52:09.613365 1751 pending_rounds.cc:69] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558376364716032 op_type: NO_OP noop_request { }
I20250115 14:52:09.613476 1751 raft_consensus.cc:2883] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:09.613555 1751 raft_consensus.cc:2267] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.615934 1751 ts_tablet_manager.cc:1905] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.617340 2424 mvcc.cc:204] Tried to move back new op lower bound from 7114558380373045248 to 7114558376364716032. Current Snapshot: MvccSnapshot[applied={T|T < 7114558380373045248}]
I20250115 14:52:09.617702 2407 tablet.cc:1721] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:09.618182 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 126e01e9520544619f3a124586937fb5 (DEFAULT_TABLE table=default.Table_8cb55ab9a193424ab91010e11b559f39 [id=22974c6d3fe74efa8ceaaba443ee0ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.618306 2273 raft_consensus.cc:491] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.618521 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 126e01e9520544619f3a124586937fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.618417 2273 raft_consensus.cc:513] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.618782 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 90764f6eaaba46ee9cdc70672d9fe732 (DEFAULT_TABLE table=default.Table_e5e8956be33c4ba7ab1e001e9719a2ef [id=45a713c968e54c7c9ca4d2dc50947a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.619011 2273 leader_election.cc:290] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.619503 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b82924e735846b2914520684d1b8fc7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:09.619617 879 tablet_service.cc:1467] Processing CreateTablet for tablet 92eaeaf2dbd84a49b704e923743e07d0 (DEFAULT_TABLE table=default.Table_01c14574e8c94e2aac622d622f8ba52c [id=2b2a775101124154bc8a37930c2a67d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.619942 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92eaeaf2dbd84a49b704e923743e07d0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:09.620087 983 leader_election.cc:343] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.620678 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b82924e735846b2914520684d1b8fc7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:09.621393 984 leader_election.cc:343] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.621486 984 leader_election.cc:304] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.621681 2273 raft_consensus.cc:2743] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:09.623926 847 consensus_peers.cc:487] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:09.624378 2403 consensus_queue.cc:1035] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:09.626403 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90764f6eaaba46ee9cdc70672d9fe732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.627278 1751 ts_tablet_manager.cc:1918] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:09.627377 1751 log.cc:1198] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/be2fd31f74a94787a5e8ccae3c5ef245
I20250115 14:52:09.627660 1751 ts_tablet_manager.cc:1939] T be2fd31f74a94787a5e8ccae3c5ef245 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.628412 1751 tablet_replica.cc:331] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.628587 1751 raft_consensus.cc:2238] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.628754 1751 raft_consensus.cc:2267] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.629392 1751 ts_tablet_manager.cc:1905] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.632237 876 tablet_service.cc:1467] Processing CreateTablet for tablet 126e01e9520544619f3a124586937fb5 (DEFAULT_TABLE table=default.Table_8cb55ab9a193424ab91010e11b559f39 [id=22974c6d3fe74efa8ceaaba443ee0ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.632790 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 126e01e9520544619f3a124586937fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.634635 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 6c028f63f7774a408a44f84e4581a753 (DEFAULT_TABLE table=default.Table_33efb40a15454adfb022b1d76c2aa409 [id=60be0b401aa14585844ca280709a355b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.636799 875 tablet_service.cc:1467] Processing CreateTablet for tablet 90764f6eaaba46ee9cdc70672d9fe732 (DEFAULT_TABLE table=default.Table_e5e8956be33c4ba7ab1e001e9719a2ef [id=45a713c968e54c7c9ca4d2dc50947a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.637228 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90764f6eaaba46ee9cdc70672d9fe732. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:09.637590 847 consensus_peers.cc:487] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80c75a77d7154b5cb05a5d9ac8a20787. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:09.647382 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 92eaeaf2dbd84a49b704e923743e07d0 (DEFAULT_TABLE table=default.Table_01c14574e8c94e2aac622d622f8ba52c [id=2b2a775101124154bc8a37930c2a67d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.648238 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet be2fd31f74a94787a5e8ccae3c5ef245 (table default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec [id=724c015c8f3f4aa581ac6faa84187b06]) successfully deleted
I20250115 14:52:09.651052 1018 tablet_service.cc:1467] Processing CreateTablet for tablet bb859e2a553f4ab3b20452b48e933f0e (DEFAULT_TABLE table=default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:09.651475 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e7315a41489a4f8296ad8b12c039b0a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7315a41489a4f8296ad8b12c039b0a2 already in progress: creating tablet
I20250115 14:52:09.590725 1164 tablet_bootstrap.cc:492] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:09.659827 1164 tablet_bootstrap.cc:654] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.662375 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92eaeaf2dbd84a49b704e923743e07d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.663009 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb859e2a553f4ab3b20452b48e933f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.663585 2273 raft_consensus.cc:491] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.663705 2273 raft_consensus.cc:513] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.664765 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580742eb5dac4dba9995809890e0e315" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:09.664811 893 tablet_service.cc:1467] Processing CreateTablet for tablet de27f6c600b840c2b00a881c5ef038ae (DEFAULT_TABLE table=default.Table_50430a4659d544eba42d4e0e1cef37da [id=8c66e79985374ed182ce8097914a49d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.665164 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de27f6c600b840c2b00a881c5ef038ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:09.665305 983 leader_election.cc:343] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.665992 2273 leader_election.cc:290] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.672338 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c028f63f7774a408a44f84e4581a753. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.675098 1751 ts_tablet_manager.cc:1918] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.675206 1751 log.cc:1198] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ba596cfb3ce64718a42a9083975edc8d
I20250115 14:52:09.675487 1751 ts_tablet_manager.cc:1939] T ba596cfb3ce64718a42a9083975edc8d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.676870 1751 tablet_replica.cc:331] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.677134 1751 raft_consensus.cc:2238] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.677335 1751 raft_consensus.cc:2267] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.678056 1751 ts_tablet_manager.cc:1905] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.678129 880 tablet_service.cc:1467] Processing CreateTablet for tablet 59e5cc832ffa4740aaacd65a565420f5 (DEFAULT_TABLE table=default.Table_3962d1f342824bf990b968aeba430bff [id=709b1a8596274f538f23cf0e9007b857]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.682009 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e32df962655644d4b4a7e3ba02623ed3" } modify_external_catalogs: true
I20250115 14:52:09.682192 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e32df962655644d4b4a7e3ba02623ed3" } modify_external_catalogs: true
I20250115 14:52:09.683597 1751 ts_tablet_manager.cc:1918] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.683689 1751 log.cc:1198] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4294c49da1e044be993100b3cf5e2cea
I20250115 14:52:09.683915 1751 ts_tablet_manager.cc:1939] T 4294c49da1e044be993100b3cf5e2cea P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.684839 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ba596cfb3ce64718a42a9083975edc8d (table default.Table_e1110e8f11bb4423bcdd8fedb129e390 [id=4d98519fb7194978a2a1533d7e42f2f1]) successfully deleted
I20250115 14:52:09.686733 1162 tablet_bootstrap.cc:492] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.686872 1162 ts_tablet_manager.cc:1397] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.133s user 0.007s sys 0.005s
I20250115 14:52:09.687498 882 tablet_service.cc:1467] Processing CreateTablet for tablet 6c028f63f7774a408a44f84e4581a753 (DEFAULT_TABLE table=default.Table_33efb40a15454adfb022b1d76c2aa409 [id=60be0b401aa14585844ca280709a355b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.687557 1162 raft_consensus.cc:357] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.687745 1162 raft_consensus.cc:383] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.687814 1162 raft_consensus.cc:738] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.688014 1162 consensus_queue.cc:260] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.688535 1162 ts_tablet_manager.cc:1428] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:09.688817 1162 tablet_bootstrap.cc:492] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.688900 1751 tablet_replica.cc:331] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.689076 1751 raft_consensus.cc:2238] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.689205 1751 raft_consensus.cc:2267] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:09.685813 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c3393ed28d242838d6f2f170184f7e8 consensus state has no leader
I20250115 14:52:09.686055 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 96369c35ca0d4a62b0676cac3dc4a813
I20250115 14:52:09.689707 1751 ts_tablet_manager.cc:1905] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.689929 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 4294c49da1e044be993100b3cf5e2cea (table default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]) successfully deleted
W20250115 14:52:09.686301 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3df4c6d310d7465ca769a5b3a13b8f59 consensus state has no leader
I20250115 14:52:09.686506 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7e600acd0ab74940a14f3022dbfa747f
I20250115 14:52:09.687072 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f7ef4548b4c2429ea9bfa1fea3d621fe" } new_table_name: "default.Table_8ea6902fe95d4b84bdaf21fd1a87e38a" modify_external_catalogs: true
I20250115 14:52:09.690934 1162 tablet_bootstrap.cc:654] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.689349 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e5cc832ffa4740aaacd65a565420f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:09.687450 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0f3f6de654d4765b1d7a381f93b0c43 consensus state has no leader
W20250115 14:52:09.687695 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29adc03e38624dfc910e07de24466d1b consensus state has no leader
I20250115 14:52:09.688192 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5c0d0b9700e4413d85d5a1dbc9a23a45" } modify_external_catalogs: true
I20250115 14:52:09.692131 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5c0d0b9700e4413d85d5a1dbc9a23a45" } modify_external_catalogs: true
I20250115 14:52:09.690507 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 96369c35ca0d4a62b0676cac3dc4a813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.688856 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 338c0f6f39fd4335bbcda733b28b8bf8
I20250115 14:52:09.697734 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c028f63f7774a408a44f84e4581a753. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.666712 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580742eb5dac4dba9995809890e0e315" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:09.698585 984 leader_election.cc:343] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.698689 984 leader_election.cc:304] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:09.699026 2273 raft_consensus.cc:2743] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.671783 744 tablet_service.cc:1467] Processing CreateTablet for tablet de27f6c600b840c2b00a881c5ef038ae (DEFAULT_TABLE table=default.Table_50430a4659d544eba42d4e0e1cef37da [id=8c66e79985374ed182ce8097914a49d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.672271 746 tablet_service.cc:1467] Processing CreateTablet for tablet bb859e2a553f4ab3b20452b48e933f0e (DEFAULT_TABLE table=default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.672739 753 tablet_service.cc:1467] Processing CreateTablet for tablet 6c028f63f7774a408a44f84e4581a753 (DEFAULT_TABLE table=default.Table_33efb40a15454adfb022b1d76c2aa409 [id=60be0b401aa14585844ca280709a355b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.673264 737 tablet_service.cc:1467] Processing CreateTablet for tablet 90764f6eaaba46ee9cdc70672d9fe732 (DEFAULT_TABLE table=default.Table_e5e8956be33c4ba7ab1e001e9719a2ef [id=45a713c968e54c7c9ca4d2dc50947a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.673586 745 tablet_service.cc:1467] Processing CreateTablet for tablet 126e01e9520544619f3a124586937fb5 (DEFAULT_TABLE table=default.Table_8cb55ab9a193424ab91010e11b559f39 [id=22974c6d3fe74efa8ceaaba443ee0ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.673641 756 tablet_service.cc:1467] Processing CreateTablet for tablet 92eaeaf2dbd84a49b704e923743e07d0 (DEFAULT_TABLE table=default.Table_01c14574e8c94e2aac622d622f8ba52c [id=2b2a775101124154bc8a37930c2a67d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.674036 747 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe88a1365954ef5beb1d67b1cc4b747 (DEFAULT_TABLE table=default.Table_1032bbe4a65c4fa1b238c728e1ab2fdf [id=66df5e7a94c04262a1c7a6ddbe28f5aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.674286 751 tablet_service.cc:1467] Processing CreateTablet for tablet a9f2e5eadb54419a9071d64474073e34 (DEFAULT_TABLE table=default.Table_f2006bfc8694432482d41dd92edbf093 [id=ee6e2171cc904b0497638242d6b43b65]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.674515 755 tablet_service.cc:1467] Processing CreateTablet for tablet 59e5cc832ffa4740aaacd65a565420f5 (DEFAULT_TABLE table=default.Table_3962d1f342824bf990b968aeba430bff [id=709b1a8596274f538f23cf0e9007b857]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.687098 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99ae2f8bd8fe4e92995dddf414172f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.700315 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 126e01e9520544619f3a124586937fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.704010 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92eaeaf2dbd84a49b704e923743e07d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.704782 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c028f63f7774a408a44f84e4581a753. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.707479 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe88a1365954ef5beb1d67b1cc4b747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.708608 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb859e2a553f4ab3b20452b48e933f0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.712424 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de27f6c600b840c2b00a881c5ef038ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.721429 1751 ts_tablet_manager.cc:1918] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:09.723363 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 85797.4us
I20250115 14:52:09.724567 1162 tablet_bootstrap.cc:492] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.724697 1162 ts_tablet_manager.cc:1397] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.001s
I20250115 14:52:09.725419 1162 raft_consensus.cc:357] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:09.725525 847 consensus_peers.cc:487] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43605c995df4400e91139199c9514bfd. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:09.725603 1162 raft_consensus.cc:383] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.725654 1162 raft_consensus.cc:738] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.725836 1162 consensus_queue.cc:260] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.726403 1162 ts_tablet_manager.cc:1428] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:09.726645 1162 tablet_bootstrap.cc:492] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.727061 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 7ea8faf380974ad9888b7da3fab6bf3e from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:09.728675 1162 tablet_bootstrap.cc:654] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.729526 1751 log.cc:1198] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/076278b53996407c8a5c73a389459eaa
I20250115 14:52:09.729806 1751 ts_tablet_manager.cc:1939] T 076278b53996407c8a5c73a389459eaa P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.730683 1751 tablet_replica.cc:331] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.730846 1751 raft_consensus.cc:2238] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.730980 1751 raft_consensus.cc:2267] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.731091 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 0651be54e0b244a4bace9f47efc9b58a from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:09.731495 1751 ts_tablet_manager.cc:1905] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.739389 1167 ts_tablet_manager.cc:1918] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.741132 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 338c0f6f39fd4335bbcda733b28b8bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.741904 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90764f6eaaba46ee9cdc70672d9fe732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.748699 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59e5cc832ffa4740aaacd65a565420f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.751771 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e600acd0ab74940a14f3022dbfa747f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.752107 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 96369c35ca0d4a62b0676cac3dc4a813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.694079 748 tablet_service.cc:1467] Processing CreateTablet for tablet d90be0a60c0a4eeca3f59afa9cfa4688 (DEFAULT_TABLE table=default.Table_fd4d720185214c7cbec302eaf573e265 [id=e976991e2fdb4163a57726f3df675713]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.688843 1166 ts_tablet_manager.cc:1918] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.752525 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f2e5eadb54419a9071d64474073e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.739344 1751 ts_tablet_manager.cc:1918] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:09.757627 1751 log.cc:1198] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3f1e5d44b8064abbbf1b6419a1eca1ed
I20250115 14:52:09.757964 1751 ts_tablet_manager.cc:1939] T 3f1e5d44b8064abbbf1b6419a1eca1ed P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.750864 1162 tablet_bootstrap.cc:492] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.759055 1162 ts_tablet_manager.cc:1397] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.000s
I20250115 14:52:09.760040 1166 log.cc:1198] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/759a4ebe7b0444128ddd31a78aad463e
I20250115 14:52:09.760051 1162 raft_consensus.cc:357] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.760257 1162 raft_consensus.cc:383] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.760330 1162 raft_consensus.cc:738] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.760597 1166 ts_tablet_manager.cc:1939] T 759a4ebe7b0444128ddd31a78aad463e P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:09.760576 1162 consensus_queue.cc:260] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.760783 2404 raft_consensus.cc:491] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.760895 2404 raft_consensus.cc:513] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.761482 2404 leader_election.cc:290] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.762926 1162 ts_tablet_manager.cc:1428] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:52:09.763191 1162 tablet_bootstrap.cc:492] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:09.763607 710 leader_election.cc:343] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 5b85d63974ab418eb1921dd605fbf2b0
I20250115 14:52:09.768779 1751 tablet_replica.cc:331] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.768965 1751 raft_consensus.cc:2238] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.769129 1751 raft_consensus.cc:2267] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.769634 1751 ts_tablet_manager.cc:1905] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.774508 1167 log.cc:1198] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/8e6cc08065494d9e85eeff5044b44ba6
I20250115 14:52:09.742997 1168 tablet_bootstrap.cc:492] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:09.774858 1167 ts_tablet_manager.cc:1939] T 8e6cc08065494d9e85eeff5044b44ba6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:09.775007 1168 ts_tablet_manager.cc:1397] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.387s user 0.010s sys 0.000s
I20250115 14:52:09.775718 1168 raft_consensus.cc:357] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.775913 1168 raft_consensus.cc:383] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.775957 1168 raft_consensus.cc:738] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.776142 1168 consensus_queue.cc:260] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.776628 1168 ts_tablet_manager.cc:1428] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:09.776890 1168 tablet_bootstrap.cc:492] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:09.778508 1162 tablet_bootstrap.cc:654] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.778542 1168 tablet_bootstrap.cc:654] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.763512 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b85d63974ab418eb1921dd605fbf2b0" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:09.779990 710 leader_election.cc:343] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:09.780125 710 leader_election.cc:304] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:09.780476 2404 raft_consensus.cc:2743] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:09.781104 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 869) took 2122 ms. Trace:
I20250115 14:52:09.781661 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 96369c35ca0d4a62b0676cac3dc4a813 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.784780 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e600acd0ab74940a14f3022dbfa747f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.787091 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 338c0f6f39fd4335bbcda733b28b8bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:09.787328 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e600acd0ab74940a14f3022dbfa747f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.787345 1167 rpcz_store.cc:276] 0115 14:52:07.659047 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.659573 (+ 526us) service_pool.cc:225] Handling call
0115 14:52:09.590906 (+1931333us) ts_tablet_manager.cc:1145] Deleting tablet 8e6cc08065494d9e85eeff5044b44ba6
0115 14:52:09.590916 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:09.675807 (+ 84891us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.739286 (+ 63479us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:09.781083 (+ 41797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1931195}
I20250115 14:52:09.787700 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 338c0f6f39fd4335bbcda733b28b8bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.788161 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a000ab73dd46ac8e134784a156040a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.788977 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 4341977f363d40eeac71b7d33d1b43ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:09.789062 1167 tablet_replica.cc:331] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:09.789194 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e034d7761a5247c79414b7112a1c97b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:09.789249 1167 raft_consensus.cc:2238] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:09.789549 1167 raft_consensus.cc:2267] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.790072 1167 ts_tablet_manager.cc:1905] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.791798 876 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe88a1365954ef5beb1d67b1cc4b747 (DEFAULT_TABLE table=default.Table_1032bbe4a65c4fa1b238c728e1ab2fdf [id=66df5e7a94c04262a1c7a6ddbe28f5aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.792166 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe88a1365954ef5beb1d67b1cc4b747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.792812 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:09.795501 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe88a1365954ef5beb1d67b1cc4b747 (DEFAULT_TABLE table=default.Table_1032bbe4a65c4fa1b238c728e1ab2fdf [id=66df5e7a94c04262a1c7a6ddbe28f5aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.795859 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fe88a1365954ef5beb1d67b1cc4b747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.802475 1751 ts_tablet_manager.cc:1918] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:09.802585 1751 log.cc:1198] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/30f3a87bf8b6452d8afd8dfd836ee4e6
I20250115 14:52:09.802903 1751 ts_tablet_manager.cc:1939] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.815829 1751 tablet_replica.cc:331] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.816051 1751 raft_consensus.cc:2238] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.816213 1751 raft_consensus.cc:2267] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.816764 1751 ts_tablet_manager.cc:1905] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.823096 876 tablet_service.cc:1467] Processing CreateTablet for tablet 068771fee92f403d8997646291a83dc6 (DEFAULT_TABLE table=default.Table_06fdf72400894754b2da4a05ae517fac [id=fa2c05b8018545958699fafc77f6e832]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.826403 882 tablet_service.cc:1467] Processing CreateTablet for tablet c300667f775e40b5a65dce61f4863cae (DEFAULT_TABLE table=default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3 [id=e9499f3a239b4bcbb1485c11a2f315c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.827839 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068771fee92f403d8997646291a83dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.828025 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d90be0a60c0a4eeca3f59afa9cfa4688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.828212 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c300667f775e40b5a65dce61f4863cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.829432 1024 tablet_service.cc:1467] Processing CreateTablet for tablet c300667f775e40b5a65dce61f4863cae (DEFAULT_TABLE table=default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3 [id=e9499f3a239b4bcbb1485c11a2f315c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.830164 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c300667f775e40b5a65dce61f4863cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.831368 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-task__Z8ae/build/asan/bin/kudu pid 1221
I20250115 14:52:09.841396 1166 tablet_replica.cc:331] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:09.841871 1166 raft_consensus.cc:2238] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.842253 1166 raft_consensus.cc:2267] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.843274 1166 ts_tablet_manager.cc:1905] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.846014 1162 tablet_bootstrap.cc:492] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.846176 1162 ts_tablet_manager.cc:1397] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.007s
I20250115 14:52:09.846949 1162 raft_consensus.cc:357] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.847157 1162 raft_consensus.cc:383] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.847220 1162 raft_consensus.cc:738] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.847461 1162 consensus_queue.cc:260] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.847960 1162 ts_tablet_manager.cc:1428] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:09.848657 1162 tablet_bootstrap.cc:492] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.850260 1162 tablet_bootstrap.cc:654] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.854988 1164 tablet_bootstrap.cc:492] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:09.855195 1164 ts_tablet_manager.cc:1397] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.266s user 0.002s sys 0.001s
I20250115 14:52:09.856940 1164 raft_consensus.cc:357] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.857331 1164 raft_consensus.cc:383] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.857483 1164 raft_consensus.cc:738] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.857947 1164 consensus_queue.cc:260] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.861802 1751 ts_tablet_manager.cc:1918] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.861917 1751 log.cc:1198] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b803d8e103a5423f85c75c9006809b17
I20250115 14:52:09.862191 1751 ts_tablet_manager.cc:1939] T b803d8e103a5423f85c75c9006809b17 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.864452 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 759a4ebe7b0444128ddd31a78aad463e (table default.Table_d1345e860a054b708e11f2f64347a3d7 [id=87e2283ba18a4525ab55a8316238aac7]) successfully deleted
W20250115 14:52:09.864940 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b3c31e24081d4a42aa933acbb80c6495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3c31e24081d4a42aa933acbb80c6495 already in progress: creating tablet
I20250115 14:52:09.867228 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b72e84abd6a046eeb0e832a67bc06490"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.867798 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b72e84abd6a046eeb0e832a67bc06490 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.868975 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ff1ff976e022472086856f885bb1f290"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.869462 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff1ff976e022472086856f885bb1f290 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.870362 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ac99de587ebb4dad9590199c16222c21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:09.870815 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac99de587ebb4dad9590199c16222c21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:09.873818 1751 tablet_replica.cc:331] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.874011 1751 raft_consensus.cc:2238] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.874153 1751 raft_consensus.cc:2267] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.874630 1751 ts_tablet_manager.cc:1905] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.881142 1162 tablet_bootstrap.cc:492] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.881350 1162 ts_tablet_manager.cc:1397] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250115 14:52:09.882135 1162 raft_consensus.cc:357] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.882588 1162 raft_consensus.cc:383] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.882685 1162 raft_consensus.cc:738] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.882969 1162 consensus_queue.cc:260] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.885432 1162 ts_tablet_manager.cc:1428] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:09.885185 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 068771fee92f403d8997646291a83dc6 (DEFAULT_TABLE table=default.Table_06fdf72400894754b2da4a05ae517fac [id=fa2c05b8018545958699fafc77f6e832]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.885761 1162 tablet_bootstrap.cc:492] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.886314 1168 tablet_bootstrap.cc:492] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:09.886503 1168 ts_tablet_manager.cc:1397] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.006s
I20250115 14:52:09.886757 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068771fee92f403d8997646291a83dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.887310 1168 raft_consensus.cc:357] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.887455 1162 tablet_bootstrap.cc:654] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.887497 1168 raft_consensus.cc:383] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.887549 1168 raft_consensus.cc:738] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.887737 1168 consensus_queue.cc:260] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.888231 1168 ts_tablet_manager.cc:1428] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:09.893569 1168 tablet_bootstrap.cc:492] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:09.894284 1019 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0bb374dc54533b7e8a5e5369c2f39 (DEFAULT_TABLE table=default.Table_0645565e1b904cff8d1ed879748d01bd [id=416f52de7ef64b15afc8a47dcb2090bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.894646 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3b0bb374dc54533b7e8a5e5369c2f39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.897529 876 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0bb374dc54533b7e8a5e5369c2f39 (DEFAULT_TABLE table=default.Table_0645565e1b904cff8d1ed879748d01bd [id=416f52de7ef64b15afc8a47dcb2090bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:09.898155 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3b0bb374dc54533b7e8a5e5369c2f39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:09.899789 1751 ts_tablet_manager.cc:1918] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.899896 1751 log.cc:1198] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6ce498ffd3f04c5d8eb642739a838903
I20250115 14:52:09.900172 1751 ts_tablet_manager.cc:1939] T 6ce498ffd3f04c5d8eb642739a838903 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.912580 1164 ts_tablet_manager.cc:1428] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.057s user 0.001s sys 0.001s
I20250115 14:52:09.920490 1168 tablet_bootstrap.cc:654] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.923508 2341 raft_consensus.cc:491] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.923636 2341 raft_consensus.cc:513] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:09.921515 1162 tablet_bootstrap.cc:492] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.924243 2341 leader_election.cc:290] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:09.924335 1162 ts_tablet_manager.cc:1397] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20250115 14:52:09.925117 1162 raft_consensus.cc:357] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.926898 1162 raft_consensus.cc:383] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.926997 1162 raft_consensus.cc:738] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.927265 1162 consensus_queue.cc:260] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:09.927564 847 leader_election.cc:343] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 07a000ab73dd46ac8e134784a156040a
I20250115 14:52:09.930410 1162 ts_tablet_manager.cc:1428] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250115 14:52:09.930727 1162 tablet_bootstrap.cc:492] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.933609 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 8e6cc08065494d9e85eeff5044b44ba6 (table default.Table_4cfdadf8899b4ab9b5400953ec24e992 [id=a990cb4c02be4fffbf1393f95c61b457]) successfully deleted
I20250115 14:52:09.938596 1162 tablet_bootstrap.cc:654] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.940682 1164 tablet_bootstrap.cc:492] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:09.939972 1751 tablet_replica.cc:331] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:09.941318 1751 raft_consensus.cc:2238] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:09.942176 1164 tablet_bootstrap.cc:654] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.945453 1751 raft_consensus.cc:2267] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:09.946014 1751 ts_tablet_manager.cc:1905] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:09.952212 1164 tablet_bootstrap.cc:492] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:09.952524 1164 ts_tablet_manager.cc:1397] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20250115 14:52:09.953768 1164 raft_consensus.cc:357] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.954157 1164 raft_consensus.cc:383] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.954298 1164 raft_consensus.cc:738] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.954700 1164 consensus_queue.cc:260] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.958498 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 076278b53996407c8a5c73a389459eaa (table default.Table_7a794e73718b4f888c401b3038c977d9 [id=2b96b49ab0d64a7db19e0fb7985b7bec]) successfully deleted
I20250115 14:52:09.959234 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3f1e5d44b8064abbbf1b6419a1eca1ed (table default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 [id=cdf45f410fda46f29cd6dbfd1cc45f28]) successfully deleted
I20250115 14:52:09.960255 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_062a9fcca27a4d90b73742167b635ae8 [id=5d05be5a21a44fe0b75c284bf3c83625]) successfully deleted
I20250115 14:52:09.969310 1162 tablet_bootstrap.cc:492] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:09.969303 1166 ts_tablet_manager.cc:1918] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.969409 1166 log.cc:1198] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/5b85d63974ab418eb1921dd605fbf2b0
I20250115 14:52:09.969697 1162 ts_tablet_manager.cc:1397] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.001s
I20250115 14:52:09.970456 1162 raft_consensus.cc:357] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.970695 1162 raft_consensus.cc:383] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:09.970706 2403 raft_consensus.cc:491] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:09.970782 1162 raft_consensus.cc:738] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:09.970845 2403 raft_consensus.cc:513] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.970996 1162 consensus_queue.cc:260] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:09.971391 2403 leader_election.cc:290] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:09.973439 1166 ts_tablet_manager.cc:1939] T 5b85d63974ab418eb1921dd605fbf2b0 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:09.975279 1162 ts_tablet_manager.cc:1428] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:09.975550 1162 tablet_bootstrap.cc:492] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:09.980574 1162 tablet_bootstrap.cc:654] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:09.982865 1164 ts_tablet_manager.cc:1428] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20250115 14:52:09.991403 1164 tablet_bootstrap.cc:492] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:09.983004 1751 ts_tablet_manager.cc:1918] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:09.995599 1751 log.cc:1198] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/580742eb5dac4dba9995809890e0e315
I20250115 14:52:09.996881 1751 ts_tablet_manager.cc:1939] T 580742eb5dac4dba9995809890e0e315 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:09.996671 1166 tablet_replica.cc:331] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:09.997349 1168 tablet_bootstrap.cc:492] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:09.997646 1168 ts_tablet_manager.cc:1397] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.004s
I20250115 14:52:09.998696 1166 raft_consensus.cc:2238] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:10.001044 1166 pending_rounds.cc:62] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:10.000447 1168 raft_consensus.cc:357] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.001788 1751 tablet_replica.cc:331] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:10.002025 1751 raft_consensus.cc:2238] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.001785 1168 raft_consensus.cc:383] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.001832 1166 pending_rounds.cc:69] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558379453489152 op_type: NO_OP noop_request { }
I20250115 14:52:10.002269 1168 raft_consensus.cc:738] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.003571 1751 raft_consensus.cc:2267] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.003037 1166 raft_consensus.cc:2883] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:10.003724 1168 consensus_queue.cc:260] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.004920 1751 ts_tablet_manager.cc:1905] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.004196 1166 raft_consensus.cc:2267] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.004441 1162 tablet_bootstrap.cc:492] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.008468 1162 ts_tablet_manager.cc:1397] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.003s
I20250115 14:52:10.009336 1162 raft_consensus.cc:357] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.009563 1162 raft_consensus.cc:383] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.009644 1162 raft_consensus.cc:738] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.009892 1162 consensus_queue.cc:260] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.010535 1162 ts_tablet_manager.cc:1428] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:10.011183 1164 tablet_bootstrap.cc:654] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.012543 1162 tablet_bootstrap.cc:492] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:10.015377 1166 ts_tablet_manager.cc:1905] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.016589 1162 tablet_bootstrap.cc:654] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.017838 771 raft_consensus.cc:3054] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:10.031842 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a000ab73dd46ac8e134784a156040a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:10.032977 748 tablet_service.cc:1467] Processing CreateTablet for tablet 068771fee92f403d8997646291a83dc6 (DEFAULT_TABLE table=default.Table_06fdf72400894754b2da4a05ae517fac [id=fa2c05b8018545958699fafc77f6e832]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.033367 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068771fee92f403d8997646291a83dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.039814 748 tablet_service.cc:1467] Processing CreateTablet for tablet c300667f775e40b5a65dce61f4863cae (DEFAULT_TABLE table=default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3 [id=e9499f3a239b4bcbb1485c11a2f315c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.040196 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c300667f775e40b5a65dce61f4863cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.042946 751 tablet_service.cc:1467] Processing CreateTablet for tablet a3b0bb374dc54533b7e8a5e5369c2f39 (DEFAULT_TABLE table=default.Table_0645565e1b904cff8d1ed879748d01bd [id=416f52de7ef64b15afc8a47dcb2090bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.043289 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3b0bb374dc54533b7e8a5e5369c2f39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.050554 1162 tablet_bootstrap.cc:492] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.050770 1162 ts_tablet_manager.cc:1397] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.001s
I20250115 14:52:10.051721 2273 raft_consensus.cc:491] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.051826 2273 raft_consensus.cc:513] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.052423 2273 leader_election.cc:290] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.053476 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48227ec5aa846c3ac63bd29ed38be25" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:10.054000 984 leader_election.cc:343] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.054785 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0d7e33fc39d94c6286060f1e02f02372
I20250115 14:52:10.055529 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4a67fd289e384cc588d7d1ae3f0ed067"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.056066 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a67fd289e384cc588d7d1ae3f0ed067 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.060209 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:10.061761 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c375b8b1b194457eb23f6594d4945a66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.062289 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c375b8b1b194457eb23f6594d4945a66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.062898 710 leader_election.cc:343] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e034d7761a5247c79414b7112a1c97b2
I20250115 14:52:10.064034 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0651be54e0b244a4bace9f47efc9b58a
I20250115 14:52:10.064532 447 catalog_manager.cc:6596] ReplaceTablet: tablet 0651be54e0b244a4bace9f47efc9b58a deleted and replaced by tablet 324c810773f249db9e8325b09805141a
I20250115 14:52:10.064937 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_da2d5c7907c24047bd4f7c0f6b70aa45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.051498 1162 raft_consensus.cc:357] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.065491 1162 raft_consensus.cc:383] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:52:10.065529 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da2d5c7907c24047bd4f7c0f6b70aa45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.065697 1162 raft_consensus.cc:738] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.066088 1162 consensus_queue.cc:260] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.067123 1166 ts_tablet_manager.cc:1918] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:10.069078 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7ea8faf380974ad9888b7da3fab6bf3e
I20250115 14:52:10.069531 454 catalog_manager.cc:6596] ReplaceTablet: tablet 7ea8faf380974ad9888b7da3fab6bf3e deleted and replaced by tablet 96692985a2c240099dbcb191fbd22384
I20250115 14:52:10.069923 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0ac18ecc16b2424881834e142ba358d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.070506 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ac18ecc16b2424881834e142ba358d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.072347 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet dadeb51154c947209dc1e00ebf844b74
I20250115 14:52:10.073017 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_10d8c601efc24a47b14b76858704b724"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.073591 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10d8c601efc24a47b14b76858704b724 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.075750 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:10.076421 1162 ts_tablet_manager.cc:1428] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250115 14:52:10.076824 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_05b6514d0e1a4ee2941c4f2b693bf29d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.077399 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05b6514d0e1a4ee2941c4f2b693bf29d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.079201 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b252e48108947cbbb36335499b51c92 has no consensus state
I20250115 14:52:10.079608 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_526b537d21ae4762b6881e46346414c6" } modify_external_catalogs: true
I20250115 14:52:10.079841 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_526b537d21ae4762b6881e46346414c6" } modify_external_catalogs: true
I20250115 14:52:10.081871 1162 tablet_bootstrap.cc:492] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:09.961462 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d6f2bc62621d402cacfcbe2eb74173f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6f2bc62621d402cacfcbe2eb74173f9 already in progress: creating tablet
W20250115 14:52:10.084362 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 123066us
I20250115 14:52:10.069674 1751 ts_tablet_manager.cc:1918] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:10.084616 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e185963130bf49a1af88ab9e14e8e1bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e185963130bf49a1af88ab9e14e8e1bc already in progress: creating tablet
I20250115 14:52:10.084802 1751 log.cc:1198] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3b82924e735846b2914520684d1b8fc7
I20250115 14:52:10.084800 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b803d8e103a5423f85c75c9006809b17 (table default.Table_4dff2e869f8141e09dd757d6e47118f4 [id=43852683a5a64e1b85403395fa3d210c]) successfully deleted
I20250115 14:52:10.085084 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6ce498ffd3f04c5d8eb642739a838903 (table default.Table_35aa379f66554bac80e34f11e41062c7 [id=c91058c4abd64f2fa0dadce63f6ce0f0]) successfully deleted
W20250115 14:52:10.085300 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 994fe2bae335497f8c07f50b91377074 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994fe2bae335497f8c07f50b91377074 already in progress: creating tablet
I20250115 14:52:10.085590 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 580742eb5dac4dba9995809890e0e315 (table default.Table_d8d3e6a881ce475a8f1319238ab8c610 [id=b6de0dbbced44c83af5f98a9d1020cfc]) successfully deleted
W20250115 14:52:10.090211 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 994fe2bae335497f8c07f50b91377074 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994fe2bae335497f8c07f50b91377074 already in progress: creating tablet
W20250115 14:52:10.090448 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0ee6906a03f546bbaaecea44cf0e7a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ee6906a03f546bbaaecea44cf0e7a68 already in progress: creating tablet
I20250115 14:52:10.090713 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 5b85d63974ab418eb1921dd605fbf2b0 (table default.Table_854a113be33b4b698a50ae5bd10a2fff [id=f1ce643dc59040148201fdd0c049fb20]) successfully deleted
I20250115 14:52:10.008023 1168 ts_tablet_manager.cc:1428] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.010s user 0.003s sys 0.002s
W20250115 14:52:10.059496 848 leader_election.cc:343] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.014343 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e034d7761a5247c79414b7112a1c97b2" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:10.093681 848 leader_election.cc:304] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.052949 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48227ec5aa846c3ac63bd29ed38be25" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:10.093731 911 raft_consensus.cc:2463] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:10.074950 1164 tablet_bootstrap.cc:492] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.090641 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d7e33fc39d94c6286060f1e02f02372 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.094179 2340 raft_consensus.cc:2743] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.094195 1164 ts_tablet_manager.cc:1397] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.002s
I20250115 14:52:10.094918 1164 raft_consensus.cc:357] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.095155 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d7e33fc39d94c6286060f1e02f02372 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.095333 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0651be54e0b244a4bace9f47efc9b58a with delete_type TABLET_DATA_DELETED (replaced by tablet 324c810773f249db9e8325b09805141a) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.095484 1164 raft_consensus.cc:383] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.095568 1164 raft_consensus.cc:738] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.095815 1164 consensus_queue.cc:260] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.090947 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0651be54e0b244a4bace9f47efc9b58a with delete_type TABLET_DATA_DELETED (replaced by tablet 324c810773f249db9e8325b09805141a) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.091080 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea8faf380974ad9888b7da3fab6bf3e with delete_type TABLET_DATA_DELETED (replaced by tablet 96692985a2c240099dbcb191fbd22384) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.091101 752 tablet_service.cc:1514] Processing DeleteTablet for tablet dadeb51154c947209dc1e00ebf844b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.097752 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea8faf380974ad9888b7da3fab6bf3e with delete_type TABLET_DATA_DELETED (replaced by tablet 96692985a2c240099dbcb191fbd22384) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.098150 876 tablet_service.cc:1514] Processing DeleteTablet for tablet dadeb51154c947209dc1e00ebf844b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.093526 1168 tablet_bootstrap.cc:492] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.098932 1164 ts_tablet_manager.cc:1428] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:10.099200 1164 tablet_bootstrap.cc:492] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.102373 1164 tablet_bootstrap.cc:654] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.104431 1166 log.cc:1198] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d9ce65604c294021af93632ed073f627
I20250115 14:52:10.104856 1166 ts_tablet_manager.cc:1939] T d9ce65604c294021af93632ed073f627 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.086558 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 65a6f83730f74d13823bd15d93538df7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.086695 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea8faf380974ad9888b7da3fab6bf3e with delete_type TABLET_DATA_DELETED (replaced by tablet 96692985a2c240099dbcb191fbd22384) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.086760 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 0651be54e0b244a4bace9f47efc9b58a with delete_type TABLET_DATA_DELETED (replaced by tablet 324c810773f249db9e8325b09805141a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.086808 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet dadeb51154c947209dc1e00ebf844b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.086877 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet be2fd31f74a94787a5e8ccae3c5ef245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.106277 710 leader_election.cc:304] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:10.108834 2403 raft_consensus.cc:2798] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:10.108906 2403 raft_consensus.cc:491] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:10.108958 2403 raft_consensus.cc:3054] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:10.109422 1751 ts_tablet_manager.cc:1939] T 3b82924e735846b2914520684d1b8fc7 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:10.109478 1166 tablet_replica.cc:331] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.109647 1166 raft_consensus.cc:2238] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.109807 1166 raft_consensus.cc:2267] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.110149 1168 tablet_bootstrap.cc:654] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.110297 1166 ts_tablet_manager.cc:1905] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.110663 1751 tablet_replica.cc:331] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:10.092975 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 0651be54e0b244a4bace9f47efc9b58a with delete_type TABLET_DATA_DELETED (replaced by tablet 324c810773f249db9e8325b09805141a) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.110826 1751 raft_consensus.cc:2238] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.093139 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d7e33fc39d94c6286060f1e02f02372 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.110972 1751 raft_consensus.cc:2267] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.111389 2403 raft_consensus.cc:513] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.112049 2403 leader_election.cc:290] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:10.094653 983 leader_election.cc:343] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.112321 983 leader_election.cc:304] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.112468 1162 tablet_bootstrap.cc:654] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.112859 2273 raft_consensus.cc:2743] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.112993 2273 raft_consensus.cc:491] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.113083 2273 raft_consensus.cc:513] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.113612 1167 ts_tablet_manager.cc:1918] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:10.113665 2273 leader_election.cc:290] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.113701 1167 log.cc:1198] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/80c75a77d7154b5cb05a5d9ac8a20787
I20250115 14:52:10.113979 1167 ts_tablet_manager.cc:1939] T 80c75a77d7154b5cb05a5d9ac8a20787 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:10.114252 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 868) took 2455 ms. Trace:
I20250115 14:52:10.114316 1167 rpcz_store.cc:276] 0115 14:52:07.658929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.659788 (+ 859us) service_pool.cc:225] Handling call
0115 14:52:09.789004 (+2129216us) ts_tablet_manager.cc:1145] Deleting tablet 80c75a77d7154b5cb05a5d9ac8a20787
0115 14:52:09.789015 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.051731 (+262716us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.113510 (+ 61779us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.114238 (+ 728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2129051}
I20250115 14:52:10.115383 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb34bf9a3464798b9f28c50d6fc11" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:10.116266 984 leader_election.cc:343] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.117262 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb34bf9a3464798b9f28c50d6fc11" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:10.117579 1164 tablet_bootstrap.cc:492] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.117681 1164 ts_tablet_manager.cc:1397] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
W20250115 14:52:10.117762 983 leader_election.cc:343] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.117859 983 leader_election.cc:304] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.118350 1164 raft_consensus.cc:357] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.118606 1164 raft_consensus.cc:383] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.118690 1164 raft_consensus.cc:738] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.118909 1164 consensus_queue.cc:260] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.119217 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e034d7761a5247c79414b7112a1c97b2" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:10.119443 913 raft_consensus.cc:3054] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:10.119513 1164 ts_tablet_manager.cc:1428] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20250115 14:52:10.120417 710 leader_election.cc:343] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e034d7761a5247c79414b7112a1c97b2
I20250115 14:52:10.120654 2469 raft_consensus.cc:2743] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.124527 1164 tablet_bootstrap.cc:492] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.125905 1164 tablet_bootstrap.cc:654] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.127748 1751 ts_tablet_manager.cc:1905] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.128654 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3b82924e735846b2914520684d1b8fc7 (table default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]) successfully deleted
W20250115 14:52:10.128954 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0ee6906a03f546bbaaecea44cf0e7a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ee6906a03f546bbaaecea44cf0e7a68 already in progress: creating tablet
W20250115 14:52:10.129160 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b3c31e24081d4a42aa933acbb80c6495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3c31e24081d4a42aa933acbb80c6495 already in progress: creating tablet
I20250115 14:52:10.129185 913 raft_consensus.cc:2463] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
W20250115 14:52:10.129386 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 96369c35ca0d4a62b0676cac3dc4a813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96369c35ca0d4a62b0676cac3dc4a813 already in progress: creating tablet
W20250115 14:52:10.129614 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7e600acd0ab74940a14f3022dbfa747f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e600acd0ab74940a14f3022dbfa747f already in progress: creating tablet
I20250115 14:52:10.129930 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 80c75a77d7154b5cb05a5d9ac8a20787 (table default.Table_5e402e29daa745aa81afd155e5c68bb4 [id=066036f5ecdf48d98dad1e1812fe55cb]) successfully deleted
I20250115 14:52:10.129918 710 leader_election.cc:304] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:10.130216 2403 raft_consensus.cc:2798] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:10.130327 2403 raft_consensus.cc:695] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:10.130540 2403 consensus_queue.cc:237] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.135285 2469 raft_consensus.cc:491] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.135479 2469 raft_consensus.cc:513] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.136492 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a6f83730f74d13823bd15d93538df7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:10.137090 984 leader_election.cc:343] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.136766 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d9ce65604c294021af93632ed073f627 (table default.Table_79fecf866fd6405a9b7df222877fc2b0 [id=0748f5a86b034a96b1dfe1cd723d2438]) successfully deleted
W20250115 14:52:10.139248 847 consensus_peers.cc:487] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f1e5d44b8064abbbf1b6419a1eca1ed. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:10.140398 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a6f83730f74d13823bd15d93538df7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:10.140914 983 leader_election.cc:343] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.141016 983 leader_election.cc:304] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.141232 2273 raft_consensus.cc:2743] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.141497 2469 leader_election.cc:290] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.144955 1166 ts_tablet_manager.cc:1918] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:10.145061 1166 log.cc:1198] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/338c0f6f39fd4335bbcda733b28b8bf8
I20250115 14:52:10.145367 1166 ts_tablet_manager.cc:1939] T 338c0f6f39fd4335bbcda733b28b8bf8 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.146144 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8f53bbc08e5340aab38bb0ce8ef7aa16" } modify_external_catalogs: true
I20250115 14:52:10.146328 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8f53bbc08e5340aab38bb0ce8ef7aa16" } modify_external_catalogs: true
I20250115 14:52:10.146822 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_25bedec15a17479282c8ddf173d89ad8" } new_table_name: "default.Table_c5ad11c078ac46fa9eecbced4da9b58e" modify_external_catalogs: true
I20250115 14:52:10.147468 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_61edbe7db3b44d02a60d60d865a86ec5" } new_table_name: "default.Table_d70b1fff9275458798bb1b6bfd566a52" modify_external_catalogs: true
I20250115 14:52:10.148016 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_29a289769ec64b96a7352fa0412f999b" } modify_external_catalogs: true
I20250115 14:52:10.148169 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_29a289769ec64b96a7352fa0412f999b" } modify_external_catalogs: true
I20250115 14:52:10.148977 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ac9b6dbf8a094c10aa1277694fc74c2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.149545 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac9b6dbf8a094c10aa1277694fc74c2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.150126 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fd4d720185214c7cbec302eaf573e265" } new_table_name: "default.Table_623f985e12f947f5b7a74539d90274f1" modify_external_catalogs: true
I20250115 14:52:10.150498 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1839edd0d49b467d8bb31b3001ff89eb
I20250115 14:52:10.151067 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_135f5999ba2b42fab38c7442ee051e44" } new_table_name: "default.Table_a8391687956e4259a71808f267cb4ff0" modify_external_catalogs: true
I20250115 14:52:10.151949 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 1839edd0d49b467d8bb31b3001ff89eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.152201 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 1839edd0d49b467d8bb31b3001ff89eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.153782 1162 tablet_bootstrap.cc:492] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.153957 1162 ts_tablet_manager.cc:1397] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20250115 14:52:10.154806 1162 raft_consensus.cc:357] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.155035 1162 raft_consensus.cc:383] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.155123 1162 raft_consensus.cc:738] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.155381 1162 consensus_queue.cc:260] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.155920 1162 ts_tablet_manager.cc:1428] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:10.157076 1168 tablet_bootstrap.cc:492] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:10.157209 1168 ts_tablet_manager.cc:1397] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250115 14:52:10.157610 1166 tablet_replica.cc:331] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.157779 1166 raft_consensus.cc:2238] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:10.158000 1166 pending_rounds.cc:62] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:10.157924 1168 raft_consensus.cc:357] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.158039 1166 pending_rounds.cc:69] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558382617067520 op_type: NO_OP noop_request { }
I20250115 14:52:10.158097 1168 raft_consensus.cc:383] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.158125 1166 raft_consensus.cc:2883] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:10.158142 1168 raft_consensus.cc:738] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.158195 1166 raft_consensus.cc:2267] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.158317 1168 consensus_queue.cc:260] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.158818 1168 ts_tablet_manager.cc:1428] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:10.159744 1166 ts_tablet_manager.cc:1905] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.156047 1751 ts_tablet_manager.cc:1918] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:10.164525 1751 log.cc:1198] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/338c0f6f39fd4335bbcda733b28b8bf8
I20250115 14:52:10.164978 1751 ts_tablet_manager.cc:1939] T 338c0f6f39fd4335bbcda733b28b8bf8 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:10.167111 848 consensus_peers.cc:487] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 338c0f6f39fd4335bbcda733b28b8bf8. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:10.167522 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 87026b6faf75437eac21dedab1f41ed4 (table default.Table_def81b30f40f4877a7e329da62ee3b27 [id=156506418e9d4322b5772bc57b409552], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 87026b6faf75437eac21dedab1f41ed4 consensus state has no leader
I20250115 14:52:10.168543 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 338c0f6f39fd4335bbcda733b28b8bf8 (table default.Table_f367ac1a82704324a73f7079165a6e45 [id=6756cde43edd4d1e8a2768c101c4aa19]) successfully deleted
W20250115 14:52:10.168889 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7e600acd0ab74940a14f3022dbfa747f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e600acd0ab74940a14f3022dbfa747f already in progress: creating tablet
W20250115 14:52:10.169070 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 96369c35ca0d4a62b0676cac3dc4a813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96369c35ca0d4a62b0676cac3dc4a813 already in progress: creating tablet
I20250115 14:52:10.170544 451 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:10.172840 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1839edd0d49b467d8bb31b3001ff89eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.172873 452 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:10.177119 1162 tablet_bootstrap.cc:492] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:10.184139 1162 tablet_bootstrap.cc:654] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.184760 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 338c0f6f39fd4335bbcda733b28b8bf8 (table default.Table_f367ac1a82704324a73f7079165a6e45 [id=6756cde43edd4d1e8a2768c101c4aa19]) successfully deleted
I20250115 14:52:10.185030 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 282d988bc41443c381bb0ba69b618f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 567365040b6d4d6eba04112c8d4c3dfb) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.185065 880 tablet_service.cc:1514] Processing DeleteTablet for tablet a600fa0f0ec049dca2803bf7d2c74a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.185437 1751 tablet_replica.cc:331] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:10.190104 1751 raft_consensus.cc:2238] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.190341 1751 raft_consensus.cc:2267] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.190924 1751 ts_tablet_manager.cc:1905] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.188230 1166 ts_tablet_manager.cc:1918] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:10.191252 1166 log.cc:1198] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e034d7761a5247c79414b7112a1c97b2
I20250115 14:52:10.191743 1166 ts_tablet_manager.cc:1939] T e034d7761a5247c79414b7112a1c97b2 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.202636 1168 tablet_bootstrap.cc:492] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.202996 1164 tablet_bootstrap.cc:492] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.203192 1164 ts_tablet_manager.cc:1397] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.010s
I20250115 14:52:10.203997 1164 raft_consensus.cc:357] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.204355 1164 raft_consensus.cc:383] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.204460 1164 raft_consensus.cc:738] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.204747 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 872) took 2534 ms. Trace:
I20250115 14:52:10.204720 1164 consensus_queue.cc:260] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.204830 1167 rpcz_store.cc:276] 0115 14:52:07.669903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.670681 (+ 778us) service_pool.cc:225] Handling call
0115 14:52:10.188414 (+2517733us) ts_tablet_manager.cc:1145] Deleting tablet b0f33b5c22ee4f8bb3ee2d20c188ade0
0115 14:52:10.204609 (+ 16195us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.204725 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2496431}
I20250115 14:52:10.205350 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 875) took 2497 ms. Trace:
I20250115 14:52:10.205430 1167 rpcz_store.cc:276] 0115 14:52:07.708098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.708403 (+ 305us) service_pool.cc:225] Handling call
0115 14:52:10.205240 (+2496837us) ts_tablet_manager.cc:1145] Deleting tablet 8f8efcf1a2db44469cd8dda0b59c34ba
0115 14:52:10.205250 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.205333 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2496589}
I20250115 14:52:10.205773 1167 tablet_replica.cc:331] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:10.206283 1167 raft_consensus.cc:2238] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.206413 1167 raft_consensus.cc:2267] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.206868 1167 ts_tablet_manager.cc:1905] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:10.209331 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0f33b5c22ee4f8bb3ee2d20c188ade0 already in progress: creating tablet
W20250115 14:52:10.209570 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 8f8efcf1a2db44469cd8dda0b59c34ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f8efcf1a2db44469cd8dda0b59c34ba already in progress: creating tablet
I20250115 14:52:10.213737 752 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ce65604c294021af93632ed073f627 with delete_type TABLET_DATA_DELETED (replaced by tablet 339dd408dab7430c980058af3af04433) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.214201 452 master_service.cc:946] ReplaceTablet: received request to replace tablet f3b75d759f834d558c94cc981e338ac0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:10.214687 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abde784f9c948cb96e6cc93c30817bd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.217607 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e034d7761a5247c79414b7112a1c97b2 (table default.Table_4deac3b7f7124b6aa49514a17811b1c3 [id=efb16fff4d6c4f44af7dfaaa00d4f444]) successfully deleted
W20250115 14:52:10.217855 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0d7e33fc39d94c6286060f1e02f02372 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d7e33fc39d94c6286060f1e02f02372 already in progress: creating tablet
W20250115 14:52:10.218070 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0651be54e0b244a4bace9f47efc9b58a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0651be54e0b244a4bace9f47efc9b58a already in progress: creating tablet
W20250115 14:52:10.218241 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7ea8faf380974ad9888b7da3fab6bf3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ea8faf380974ad9888b7da3fab6bf3e already in progress: creating tablet
I20250115 14:52:10.219808 1162 tablet_bootstrap.cc:492] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.219995 1162 ts_tablet_manager.cc:1397] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20250115 14:52:10.220822 1162 raft_consensus.cc:357] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.221071 1162 raft_consensus.cc:383] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.221167 1162 raft_consensus.cc:738] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.221424 1162 consensus_queue.cc:260] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.225927 1162 ts_tablet_manager.cc:1428] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250115 14:52:10.226260 1162 tablet_bootstrap.cc:492] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:10.227969 1162 tablet_bootstrap.cc:654] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.231637 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 377d711f052545ddbe499afe7ba3de28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.235481 1164 ts_tablet_manager.cc:1428] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20250115 14:52:10.230266 1751 ts_tablet_manager.cc:1918] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.236527 1751 log.cc:1198] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/65a6f83730f74d13823bd15d93538df7
I20250115 14:52:10.236953 1751 ts_tablet_manager.cc:1939] T 65a6f83730f74d13823bd15d93538df7 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:10.239745 1164 tablet_bootstrap.cc:492] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.240391 1168 tablet_bootstrap.cc:654] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.242008 1164 tablet_bootstrap.cc:654] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:10.243176 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 288b0e455a854464940f65a2ab909d0b consensus state has no leader
I20250115 14:52:10.243291 1166 tablet_replica.cc:331] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.248457 1166 raft_consensus.cc:2238] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.248644 1166 raft_consensus.cc:2267] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.249197 1166 ts_tablet_manager.cc:1905] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.250483 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 65a6f83730f74d13823bd15d93538df7 (table default.Table_8a0954ff11744933ba4752116830cdd7 [id=a3bf28673dac4ddab579709d6fd8885d]) successfully deleted
I20250115 14:52:10.250783 1751 tablet_replica.cc:331] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:10.250977 1751 raft_consensus.cc:2238] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.251113 1751 raft_consensus.cc:2267] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:10.250810 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7ea8faf380974ad9888b7da3fab6bf3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ea8faf380974ad9888b7da3fab6bf3e already in progress: creating tablet
I20250115 14:52:10.251639 1751 ts_tablet_manager.cc:1905] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.254665 1162 tablet_bootstrap.cc:492] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.254833 1162 ts_tablet_manager.cc:1397] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20250115 14:52:10.255532 1162 raft_consensus.cc:357] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.255801 1162 raft_consensus.cc:383] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.255890 1162 raft_consensus.cc:738] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.256134 1162 consensus_queue.cc:260] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.256670 1162 ts_tablet_manager.cc:1428] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:10.267350 1751 ts_tablet_manager.cc:1918] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.269048 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ca128bf3d6fc430298da932a78f54184
W20250115 14:52:10.269861 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edb5b4911c70478f8d02e476e9cb1769 consensus state has no leader
I20250115 14:52:10.269881 2479 tablet.cc:1721] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:10.270896 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6fd405050a2548e1b9ea7d7bf34a35af
W20250115 14:52:10.272137 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d90be0a60c0a4eeca3f59afa9cfa4688 consensus state has no leader
W20250115 14:52:10.272898 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5e97279185541b992340c24ede01656 consensus state has no leader
I20250115 14:52:10.276603 880 tablet_service.cc:1514] Processing DeleteTablet for tablet ca128bf3d6fc430298da932a78f54184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.279927 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd405050a2548e1b9ea7d7bf34a35af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:10.288074 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet dadeb51154c947209dc1e00ebf844b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dadeb51154c947209dc1e00ebf844b74 already in progress: creating tablet
I20250115 14:52:10.293895 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ca128bf3d6fc430298da932a78f54184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.268571 1162 tablet_bootstrap.cc:492] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:10.296134 1166 ts_tablet_manager.cc:1918] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:10.296242 1166 log.cc:1198] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/1839edd0d49b467d8bb31b3001ff89eb
I20250115 14:52:10.296576 1166 ts_tablet_manager.cc:1939] T 1839edd0d49b467d8bb31b3001ff89eb P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.297191 1162 tablet_bootstrap.cc:654] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.276861 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet ca128bf3d6fc430298da932a78f54184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.280493 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd405050a2548e1b9ea7d7bf34a35af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.300952 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd405050a2548e1b9ea7d7bf34a35af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.306608 1164 tablet_bootstrap.cc:492] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.306809 1164 ts_tablet_manager.cc:1397] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.004s
I20250115 14:52:10.307595 1164 raft_consensus.cc:357] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.307859 1164 raft_consensus.cc:383] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.307963 1164 raft_consensus.cc:738] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.308250 1164 consensus_queue.cc:260] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.307058 1168 tablet_bootstrap.cc:492] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:10.309788 1168 ts_tablet_manager.cc:1397] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.000s
I20250115 14:52:10.311826 1164 ts_tablet_manager.cc:1428] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250115 14:52:10.312196 1164 tablet_bootstrap.cc:492] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.312256 2480 tablet.cc:1721] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:10.314450 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f3b75d759f834d558c94cc981e338ac0
I20250115 14:52:10.314898 452 catalog_manager.cc:6596] ReplaceTablet: tablet f3b75d759f834d558c94cc981e338ac0 deleted and replaced by tablet 7539a25fbf6c4e28b5538c8a29eaf529
I20250115 14:52:10.316937 880 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b75d759f834d558c94cc981e338ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7539a25fbf6c4e28b5538c8a29eaf529) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.322296 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b75d759f834d558c94cc981e338ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7539a25fbf6c4e28b5538c8a29eaf529) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.324915 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 1839edd0d49b467d8bb31b3001ff89eb (table default.Table_526b537d21ae4762b6881e46346414c6 [id=a724f6e4ad16485b8ee2445bb072be3e]) successfully deleted
W20250115 14:52:10.325260 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d9ce65604c294021af93632ed073f627 because the tablet was not found. No further retry: Not found: Tablet not found: d9ce65604c294021af93632ed073f627
I20250115 14:52:10.326346 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_225689bca59c415d884b3d3ca93d556b" } modify_external_catalogs: true
I20250115 14:52:10.326560 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_225689bca59c415d884b3d3ca93d556b" } modify_external_catalogs: true
I20250115 14:52:10.327327 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_94471b2570804dbca5f0b06961b996e2" } modify_external_catalogs: true
I20250115 14:52:10.327494 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_94471b2570804dbca5f0b06961b996e2" } modify_external_catalogs: true
I20250115 14:52:10.328145 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0195de43d08f499d99237ab07b24b892" } new_table_name: "default.Table_b05c36755c204b1685b0d526f655cc9a" modify_external_catalogs: true
I20250115 14:52:10.331153 1162 tablet_bootstrap.cc:492] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.331357 1162 ts_tablet_manager.cc:1397] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.000s
I20250115 14:52:10.332208 1162 raft_consensus.cc:357] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.332495 1162 raft_consensus.cc:383] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.332588 1162 raft_consensus.cc:738] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.332866 1162 consensus_queue.cc:260] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:10.340332 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13053264b5ca41dd85e53d928aa511fd consensus state has no leader
I20250115 14:52:10.340673 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_11d56e79fc964adbb45d31451183c7d9 [id=8725c85d30684db495ec7a20962fd75d] alter complete (version 1)
I20250115 14:52:10.344545 1162 ts_tablet_manager.cc:1428] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20250115 14:52:10.349418 1162 tablet_bootstrap.cc:492] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:10.354884 1162 tablet_bootstrap.cc:654] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.355037 1751 log.cc:1198] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/0651be54e0b244a4bace9f47efc9b58a
I20250115 14:52:10.356160 1751 ts_tablet_manager.cc:1939] T 0651be54e0b244a4bace9f47efc9b58a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:10.310504 1168 raft_consensus.cc:357] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.357426 1168 raft_consensus.cc:383] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.357542 1168 raft_consensus.cc:738] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.358281 1168 consensus_queue.cc:260] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.359217 1168 ts_tablet_manager.cc:1428] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.049s user 0.001s sys 0.001s
I20250115 14:52:10.313740 1164 tablet_bootstrap.cc:654] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.318233 748 tablet_service.cc:1514] Processing DeleteTablet for tablet f3b75d759f834d558c94cc981e338ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet 7539a25fbf6c4e28b5538c8a29eaf529) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.320420 1166 tablet_replica.cc:331] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.321456 752 tablet_service.cc:1514] Processing DeleteTablet for tablet efb8b001dd66456d9e49e9e8a6848ccf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.321815 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 50273a0019bb4cd8977f7ef014372d85 with delete_type TABLET_DATA_DELETED (replaced by tablet 70425c25919b4d0b999450210e1cb33f) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.322199 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8efcf1a2db44469cd8dda0b59c34ba) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.361515 1166 raft_consensus.cc:2238] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.361663 1166 raft_consensus.cc:2267] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.362192 1166 ts_tablet_manager.cc:1905] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.367982 1166 ts_tablet_manager.cc:1918] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.369832 1167 ts_tablet_manager.cc:1918] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.370051 1167 log.cc:1198] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e034d7761a5247c79414b7112a1c97b2
I20250115 14:52:10.370555 1167 ts_tablet_manager.cc:1939] T e034d7761a5247c79414b7112a1c97b2 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:10.371385 1166 log.cc:1198] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/4abde784f9c948cb96e6cc93c30817bd
I20250115 14:52:10.371866 1166 ts_tablet_manager.cc:1939] T 4abde784f9c948cb96e6cc93c30817bd P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.373469 1166 tablet_replica.cc:331] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.376436 1166 raft_consensus.cc:2238] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.376665 1166 raft_consensus.cc:2267] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.377182 1168 tablet_bootstrap.cc:492] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.379101 2341 raft_consensus.cc:491] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.379225 2341 raft_consensus.cc:513] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.379824 2341 leader_election.cc:290] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:10.381165 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 880) took 2543 ms. Trace:
I20250115 14:52:10.381299 1167 rpcz_store.cc:276] 0115 14:52:07.838095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.838156 (+ 61us) service_pool.cc:225] Handling call
0115 14:52:10.205738 (+2367582us) ts_tablet_manager.cc:1145] Deleting tablet e034d7761a5247c79414b7112a1c97b2
0115 14:52:10.205746 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.259219 (+ 53473us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.369640 (+110421us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.381124 (+ 11484us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367403}
W20250115 14:52:10.381982 847 leader_election.cc:343] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 84844b5f50af408ab7dda73f7421cd9d
I20250115 14:52:10.382042 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 882) took 2543 ms. Trace:
I20250115 14:52:10.382170 1167 rpcz_store.cc:276] 0115 14:52:07.838898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.838961 (+ 63us) service_pool.cc:225] Handling call
0115 14:52:10.381846 (+2542885us) ts_tablet_manager.cc:1145] Deleting tablet 325cd3fd8b4d447998822163e575e758
0115 14:52:10.381854 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.382024 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2542716}
I20250115 14:52:10.382367 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84844b5f50af408ab7dda73f7421cd9d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:10.382756 1166 ts_tablet_manager.cc:1905] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:10.382974 848 leader_election.cc:343] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.383078 848 leader_election.cc:304] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.383175 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 883) took 2544 ms. Trace:
I20250115 14:52:10.383569 2341 raft_consensus.cc:2743] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.389951 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 4abde784f9c948cb96e6cc93c30817bd (table default.Table_10deaf92a667422fa029a731f755dfef [id=e657e05ca35045f1a196dd012ee59483]) successfully deleted
W20250115 14:52:10.390369 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ca128bf3d6fc430298da932a78f54184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca128bf3d6fc430298da932a78f54184 already in progress: creating tablet
W20250115 14:52:10.390657 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6fd405050a2548e1b9ea7d7bf34a35af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fd405050a2548e1b9ea7d7bf34a35af already in progress: creating tablet
I20250115 14:52:10.391294 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e7e9c0d93b9d4ec48187e83d5c70b6ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.392329 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_657b046561f24da2b58925ba6ef4f8ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.393337 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c7660a4693d04427a14694d941c8f631"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.393530 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f3626b709cb3478597936e4d350b38a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.394663 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f436acfa7f9d4abd8d6ca5f71c593faa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.395991 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_311231a4be29497898f249683c9fb37d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.398849 2486 tablet.cc:1721] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:10.400346 1168 tablet_bootstrap.cc:654] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.400514 1162 tablet_bootstrap.cc:492] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.401667 1162 ts_tablet_manager.cc:1397] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.001s
I20250115 14:52:10.402487 1162 raft_consensus.cc:357] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.403981 1162 raft_consensus.cc:383] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.404107 1162 raft_consensus.cc:738] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.401475 1751 tablet_replica.cc:331] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:10.404516 1751 raft_consensus.cc:2238] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:10.404610 1164 tablet_bootstrap.cc:492] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.404873 1751 raft_consensus.cc:2267] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.404935 1164 ts_tablet_manager.cc:1397] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.006s
I20250115 14:52:10.405452 1751 ts_tablet_manager.cc:1905] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.406348 1164 raft_consensus.cc:357] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.406770 1164 raft_consensus.cc:383] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.406914 1164 raft_consensus.cc:738] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.407379 1164 consensus_queue.cc:260] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.383267 1167 rpcz_store.cc:276] 0115 14:52:07.839058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:07.839845 (+ 787us) service_pool.cc:225] Handling call
0115 14:52:10.382945 (+2543100us) ts_tablet_manager.cc:1145] Deleting tablet c042a4f4f9bc46afb0b94df8ed74a8cf
0115 14:52:10.382982 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.383137 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2542943}
I20250115 14:52:10.410331 1164 ts_tablet_manager.cc:1428] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:10.410529 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 892) took 2362 ms. Trace:
I20250115 14:52:10.410841 1167 rpcz_store.cc:276] 0115 14:52:08.047865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.053580 (+ 5715us) service_pool.cc:225] Handling call
0115 14:52:10.410142 (+2356562us) ts_tablet_manager.cc:1145] Deleting tablet 2f7a5cbd9a78485a816d7fecdc23106a
0115 14:52:10.410157 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.410508 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2356280}
I20250115 14:52:10.412133 1164 tablet_bootstrap.cc:492] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.412359 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 893) took 2358 ms. Trace:
I20250115 14:52:10.412658 1167 rpcz_store.cc:276] 0115 14:52:08.053981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.058787 (+ 4806us) service_pool.cc:225] Handling call
0115 14:52:10.411988 (+2353201us) ts_tablet_manager.cc:1145] Deleting tablet 005a4cd0e9f14e76b01afa49d946ee32
0115 14:52:10.412007 (+ 19us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.412339 (+ 332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2352993}
I20250115 14:52:10.413849 1167 tablet_replica.cc:331] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:10.414572 1167 raft_consensus.cc:2238] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.415129 1167 raft_consensus.cc:2267] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:10.415550 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_311231a4be29497898f249683c9fb37d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.415624 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_657b046561f24da2b58925ba6ef4f8ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.415764 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f436acfa7f9d4abd8d6ca5f71c593faa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.415568 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7e9c0d93b9d4ec48187e83d5c70b6ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.418659 1162 consensus_queue.cc:260] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:10.419521 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3626b709cb3478597936e4d350b38a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.420059 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7660a4693d04427a14694d941c8f631 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.424541 1162 ts_tablet_manager.cc:1428] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250115 14:52:10.357143 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 0651be54e0b244a4bace9f47efc9b58a (table default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]) successfully deleted
W20250115 14:52:10.425082 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 67950.9us
I20250115 14:52:10.425413 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet e034d7761a5247c79414b7112a1c97b2 (table default.Table_4deac3b7f7124b6aa49514a17811b1c3 [id=efb16fff4d6c4f44af7dfaaa00d4f444]) successfully deleted
W20250115 14:52:10.425634 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 325cd3fd8b4d447998822163e575e758 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 325cd3fd8b4d447998822163e575e758 already in progress: creating tablet
W20250115 14:52:10.425841 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c042a4f4f9bc46afb0b94df8ed74a8cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c042a4f4f9bc46afb0b94df8ed74a8cf already in progress: creating tablet
W20250115 14:52:10.426048 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2f7a5cbd9a78485a816d7fecdc23106a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f7a5cbd9a78485a816d7fecdc23106a already in progress: creating tablet
W20250115 14:52:10.426239 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 005a4cd0e9f14e76b01afa49d946ee32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 005a4cd0e9f14e76b01afa49d946ee32 already in progress: creating tablet
W20250115 14:52:10.426424 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet dadeb51154c947209dc1e00ebf844b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dadeb51154c947209dc1e00ebf844b74 already in progress: creating tablet
W20250115 14:52:10.426626 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet be2fd31f74a94787a5e8ccae3c5ef245 because the tablet was not found. No further retry: Not found: Tablet not found: be2fd31f74a94787a5e8ccae3c5ef245
W20250115 14:52:10.426806 440 catalog_manager.cc:4839] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0651be54e0b244a4bace9f47efc9b58a because the tablet was not found. No further retry: Not found: Tablet not found: 0651be54e0b244a4bace9f47efc9b58a
W20250115 14:52:10.427023 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0d7e33fc39d94c6286060f1e02f02372 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d7e33fc39d94c6286060f1e02f02372 already in progress: creating tablet
I20250115 14:52:10.427577 1164 tablet_bootstrap.cc:654] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.427629 1162 tablet_bootstrap.cc:492] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:10.430124 1166 ts_tablet_manager.cc:1918] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:10.438664 1166 log.cc:1198] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f3b75d759f834d558c94cc981e338ac0
I20250115 14:52:10.439348 1166 ts_tablet_manager.cc:1939] T f3b75d759f834d558c94cc981e338ac0 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.443375 1164 tablet_bootstrap.cc:492] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.443398 1162 tablet_bootstrap.cc:654] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.444036 1164 ts_tablet_manager.cc:1397] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20250115 14:52:10.453841 2403 raft_consensus.cc:491] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.453984 2403 raft_consensus.cc:513] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.454638 2403 leader_election.cc:290] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.455338 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8442dc919d5f4d63b9b2ce7276cc0bc5" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:10.455550 913 raft_consensus.cc:2463] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:10.443611 1166 tablet_replica.cc:331] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.456099 710 leader_election.cc:304] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:10.456179 1166 raft_consensus.cc:2238] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.456418 1166 raft_consensus.cc:2267] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.456596 2403 raft_consensus.cc:2798] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:10.456691 2403 raft_consensus.cc:491] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:10.456758 2403 raft_consensus.cc:3054] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:10.459467 2403 raft_consensus.cc:513] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.460053 2403 leader_election.cc:290] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:10.456615 2324 raft_consensus.cc:668] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20250115 14:52:10.461862 1166 ts_tablet_manager.cc:1905] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.463922 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8442dc919d5f4d63b9b2ce7276cc0bc5" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:10.464119 913 raft_consensus.cc:3054] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:10.465051 2341 raft_consensus.cc:491] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.465193 2341 raft_consensus.cc:513] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.465756 2341 leader_election.cc:290] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:10.466532 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f3b75d759f834d558c94cc981e338ac0 (table default.Table_01bf4fd686f74e258977d847e2ee966e [id=f4f61992d591495f9e7a5a4b5c94fa32]) successfully deleted
I20250115 14:52:10.467386 913 raft_consensus.cc:2463] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:10.470494 1166 ts_tablet_manager.cc:1918] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.470665 1166 log.cc:1198] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/efb8b001dd66456d9e49e9e8a6848ccf
I20250115 14:52:10.471009 1166 ts_tablet_manager.cc:1939] T efb8b001dd66456d9e49e9e8a6848ccf P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.472955 710 leader_election.cc:304] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:10.473312 1166 tablet_replica.cc:331] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
W20250115 14:52:10.473392 710 leader_election.cc:343] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 8442dc919d5f4d63b9b2ce7276cc0bc5
I20250115 14:52:10.473503 1167 ts_tablet_manager.cc:1905] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.473534 1458 generic_service.cc:196] Checking for leaks (request via RPC)
I20250115 14:52:10.473645 1166 raft_consensus.cc:2238] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.473824 2324 raft_consensus.cc:2798] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:10.473842 1166 raft_consensus.cc:2267] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.474103 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282d988bc41443c381bb0ba69b618f5d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:10.474223 847 leader_election.cc:343] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 282d988bc41443c381bb0ba69b618f5d
I20250115 14:52:10.474303 771 raft_consensus.cc:2463] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:10.474412 1168 tablet_bootstrap.cc:492] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:10.473953 2324 raft_consensus.cc:695] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:10.474536 1168 ts_tablet_manager.cc:1397] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.097s user 0.001s sys 0.003s
I20250115 14:52:10.474673 2324 consensus_queue.cc:237] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.474802 1166 ts_tablet_manager.cc:1905] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.475253 1168 raft_consensus.cc:357] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.475457 1168 raft_consensus.cc:383] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.475502 1168 raft_consensus.cc:738] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.475693 1168 consensus_queue.cc:260] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.476321 1168 ts_tablet_manager.cc:1428] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:10.476986 1168 tablet_bootstrap.cc:492] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.477250 848 leader_election.cc:304] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:10.477423 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet efb8b001dd66456d9e49e9e8a6848ccf (table default.Table_e8f8acbb30db4f019ebeadda783b434a [id=877cb5d7df394006822f9a0e8a133dcc]) successfully deleted
I20250115 14:52:10.477629 2341 raft_consensus.cc:2798] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:10.477702 2341 raft_consensus.cc:491] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:10.477761 2341 raft_consensus.cc:3054] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:10.479867 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a18f52945a28410fb91e6dfeef1cdcb9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.480888 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_958dafb4bff949ae8e00c6981747aa85" } modify_external_catalogs: true
I20250115 14:52:10.481094 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_958dafb4bff949ae8e00c6981747aa85" } modify_external_catalogs: true
I20250115 14:52:10.481142 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e55f8e94ee304fd9ab86502d92eecf50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.482966 2341 raft_consensus.cc:513] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.483621 2341 leader_election.cc:290] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:10.483956 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a18f52945a28410fb91e6dfeef1cdcb9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.484413 847 leader_election.cc:343] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 282d988bc41443c381bb0ba69b618f5d
W20250115 14:52:10.489120 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e55f8e94ee304fd9ab86502d92eecf50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.491393 755 tablet_service.cc:1514] Processing DeleteTablet for tablet efb8b001dd66456d9e49e9e8a6848ccf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.491621 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e95d09efd2e44cc81cdcbf87ee453fe with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd94e4a7fef44f5be462e7a77e8a123) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.493172 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "282d988bc41443c381bb0ba69b618f5d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:10.493358 771 raft_consensus.cc:3054] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:10.495745 1168 tablet_bootstrap.cc:654] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.496024 771 raft_consensus.cc:2463] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:10.444957 1164 raft_consensus.cc:357] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.497037 1164 raft_consensus.cc:383] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.497066 848 leader_election.cc:304] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:10.497129 1164 raft_consensus.cc:738] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.497341 1164 consensus_queue.cc:260] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.497869 1166 ts_tablet_manager.cc:1918] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.498028 1166 log.cc:1198] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/50273a0019bb4cd8977f7ef014372d85
I20250115 14:52:10.498343 1166 ts_tablet_manager.cc:1939] T 50273a0019bb4cd8977f7ef014372d85 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.499537 1166 tablet_replica.cc:331] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.501027 2341 raft_consensus.cc:2798] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:10.501142 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 50273a0019bb4cd8977f7ef014372d85 (table default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]) successfully deleted
I20250115 14:52:10.501361 2341 raft_consensus.cc:695] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:10.501900 2341 consensus_queue.cc:237] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.504789 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5b2238ba86fe41079543db9d13c4c1f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.505316 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b2238ba86fe41079543db9d13c4c1f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.508416 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_a9098644161a45d4be352eb0b77e7eca" } new_table_name: "default.Table_5ff3a565903a49c4b0d981883ca6f331" modify_external_catalogs: true
W20250115 14:52:10.512460 710 leader_election.cc:343] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 8442dc919d5f4d63b9b2ce7276cc0bc5
I20250115 14:52:10.513916 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 5265d5eeac83481ca97801e3475886da from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:10.514430 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b05c36755c204b1685b0d526f655cc9a" } modify_external_catalogs: true
I20250115 14:52:10.514581 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b05c36755c204b1685b0d526f655cc9a" } modify_external_catalogs: true
I20250115 14:52:10.516709 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 282d988bc41443c381bb0ba69b618f5d with delete_type TABLET_DATA_DELETED (replaced by tablet 567365040b6d4d6eba04112c8d4c3dfb) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.518584 1164 ts_tablet_manager.cc:1428] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.074s user 0.000s sys 0.001s
I20250115 14:52:10.522349 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_03c9410818db4ce190e6ca259f472d70" } new_table_name: "default.Table_daaa2755fe9d4f54ae6ff33ae452ec05" modify_external_catalogs: true
I20250115 14:52:10.523690 1164 tablet_bootstrap.cc:492] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.523782 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ff1ff976e022472086856f885bb1f290" } modify_external_catalogs: true
I20250115 14:52:10.523922 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ff1ff976e022472086856f885bb1f290" } modify_external_catalogs: true
I20250115 14:52:10.525876 1164 tablet_bootstrap.cc:654] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.527267 1167 ts_tablet_manager.cc:1918] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.528462 1167 log.cc:1198] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/3a0d97fa57484b9d8cb879fedda61612
I20250115 14:52:10.529620 1167 ts_tablet_manager.cc:1939] T 3a0d97fa57484b9d8cb879fedda61612 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:10.530197 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 894) took 2471 ms. Trace:
I20250115 14:52:10.532177 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b3e4daa02d5e4d8ca7f0710e6c5f8260" } new_table_name: "default.Table_35a3371b3a6c4f35b2990f828671681e" modify_external_catalogs: true
I20250115 14:52:10.534545 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 2484885df0da46f381d625a26e211c1e (table default.Table_8652d6d9ec084553b5183aa372c2554a [id=c29c4105b83d4df9b617997c0c9c5b14]). Expected version 1 got 0
I20250115 14:52:10.535266 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_00e45525259140518db638102c114d02" } new_table_name: "default.Table_760a381e30f24e3bb11cf7bb37c10d75" modify_external_catalogs: true
I20250115 14:52:10.534571 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c1102208539341c0a46b3222273132f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.536850 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1102208539341c0a46b3222273132f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.541603 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
I20250115 14:52:10.541812 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]). Expected version 1 got 0
I20250115 14:52:10.542184 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 5df0250d08214a9783c392071126ed4a (table default.Table_90c834d5ef634e6aa797dd220e3d5faa [id=8ffbe34c2a11417f8ca50dbdd627bf22]). Expected version 2 got 0
I20250115 14:52:10.546453 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet a48227ec5aa846c3ac63bd29ed38be25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.546674 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e616d78933e4f84b9ec2acc51450b18 with delete_type TABLET_DATA_DELETED (replaced by tablet ba33eb3f5a40450db5cf115fbe8a17e7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.546849 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.548358 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fe51b51994491ab38a644bfc55e3ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.548883 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 005a4cd0e9f14e76b01afa49d946ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.550431 451 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 2a64117652414d92a7b87019dd846933
I20250115 14:52:10.549461 880 tablet_service.cc:1467] Processing CreateTablet for tablet a3b82825622b45c7aa1f076e83c1ae78 (DEFAULT_TABLE table=default.Table_1b68293cce4a49c3b939caef23d5e36c [id=588269769c5f4f5da4d677e56daf1363]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.551178 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3b82825622b45c7aa1f076e83c1ae78. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:10.554229 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 246720dbb4ed4216ac87fb4a3df79e2c consensus state has no leader
I20250115 14:52:10.554908 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_657b046561f24da2b58925ba6ef4f8ba" } new_table_name: "default.Table_4a7ea7c535504cfdb8ba930aa3f60055" modify_external_catalogs: true
I20250115 14:52:10.556423 753 tablet_service.cc:1467] Processing CreateTablet for tablet a3b82825622b45c7aa1f076e83c1ae78 (DEFAULT_TABLE table=default.Table_1b68293cce4a49c3b939caef23d5e36c [id=588269769c5f4f5da4d677e56daf1363]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.549526 1843 raft_consensus.cc:491] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.556815 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3b82825622b45c7aa1f076e83c1ae78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.556840 1843 raft_consensus.cc:513] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.556946 1166 raft_consensus.cc:2238] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:10.557276 1166 pending_rounds.cc:62] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:10.557322 1166 pending_rounds.cc:69] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558384026791936 op_type: NO_OP noop_request { }
I20250115 14:52:10.557407 1166 raft_consensus.cc:2883] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:10.557430 1843 leader_election.cc:290] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:10.557480 1166 raft_consensus.cc:2267] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.558430 2469 raft_consensus.cc:491] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:52:10.558524 847 leader_election.cc:343] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: a600fa0f0ec049dca2803bf7d2c74a95
I20250115 14:52:10.558552 2469 raft_consensus.cc:513] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.559113 2469 leader_election.cc:290] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.559870 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005a4cd0e9f14e76b01afa49d946ee32" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:10.560551 984 leader_election.cc:343] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.561050 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a600fa0f0ec049dca2803bf7d2c74a95" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:10.561203 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005a4cd0e9f14e76b01afa49d946ee32" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:10.561683 983 leader_election.cc:343] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.561794 983 leader_election.cc:304] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.562132 2273 raft_consensus.cc:2743] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:10.563879 848 leader_election.cc:343] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.563985 848 leader_election.cc:304] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.565306 1843 raft_consensus.cc:2743] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.550137 1168 tablet_bootstrap.cc:492] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:10.565560 1168 ts_tablet_manager.cc:1397] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.002s
I20250115 14:52:10.566298 1168 raft_consensus.cc:357] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.566483 1168 raft_consensus.cc:383] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.566548 1168 raft_consensus.cc:738] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.566721 1168 consensus_queue.cc:260] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.567239 1168 ts_tablet_manager.cc:1428] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:10.567484 1168 tablet_bootstrap.cc:492] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.569514 1168 tablet_bootstrap.cc:654] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.571446 753 tablet_service.cc:1514] Processing DeleteTablet for tablet 90c1602aef514dd29d5d2cffb043dbb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.571610 1167 rpcz_store.cc:276] 0115 14:52:08.059152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.060897 (+ 1745us) service_pool.cc:225] Handling call
0115 14:52:10.413703 (+2352806us) ts_tablet_manager.cc:1145] Deleting tablet 3a0d97fa57484b9d8cb879fedda61612
0115 14:52:10.413762 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.490684 (+ 76922us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.527151 (+ 36467us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.530179 (+ 3028us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2351089}
I20250115 14:52:10.571993 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 3a0d97fa57484b9d8cb879fedda61612 (table default.Table_de1856a934f84a1fa118e04a42ef373e [id=55e89953f6914380bb5b2d81aa9da51b]) successfully deleted
I20250115 14:52:10.572324 455 master_service.cc:946] ReplaceTablet: received request to replace tablet c6f9262e808b4589a21257ed3fb0363b from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:10.574959 2479 tablet.cc:1721] T df84bc24a71e431cb47b93f2b0dcaa90 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:10.576381 2486 tablet.cc:1721] T df84bc24a71e431cb47b93f2b0dcaa90 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:10.578855 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 896) took 2493 ms. Trace:
I20250115 14:52:10.578929 1167 rpcz_store.cc:276] 0115 14:52:08.085857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.086311 (+ 454us) service_pool.cc:225] Handling call
0115 14:52:10.578755 (+2492444us) ts_tablet_manager.cc:1145] Deleting tablet e910c387e6a24caf95e14b57c8b83e6f
0115 14:52:10.578763 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.578839 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2492272}
I20250115 14:52:10.579236 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 897) took 2492 ms. Trace:
W20250115 14:52:10.579327 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e910c387e6a24caf95e14b57c8b83e6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e910c387e6a24caf95e14b57c8b83e6f already in progress: creating tablet
I20250115 14:52:10.579308 1167 rpcz_store.cc:276] 0115 14:52:08.086733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.086886 (+ 153us) service_pool.cc:225] Handling call
0115 14:52:10.579139 (+2492253us) ts_tablet_manager.cc:1145] Deleting tablet b54807a4e9644b86a89670f57ae154a3
0115 14:52:10.579146 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.579222 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2492049}
I20250115 14:52:10.579587 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 898) took 2386 ms. Trace:
W20250115 14:52:10.579706 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b54807a4e9644b86a89670f57ae154a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b54807a4e9644b86a89670f57ae154a3 already in progress: creating tablet
I20250115 14:52:10.579669 1167 rpcz_store.cc:276] 0115 14:52:08.193586 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.193635 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:10.579503 (+2385868us) ts_tablet_manager.cc:1145] Deleting tablet d360add9d39a48caa7b90f7d786f7bc1
0115 14:52:10.579511 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.579574 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373260}
I20250115 14:52:10.579964 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 899) took 2383 ms. Trace:
W20250115 14:52:10.580085 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d360add9d39a48caa7b90f7d786f7bc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d360add9d39a48caa7b90f7d786f7bc1 already in progress: creating tablet
I20250115 14:52:10.580049 1167 rpcz_store.cc:276] 0115 14:52:08.196829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.196877 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:10.579876 (+2382999us) ts_tablet_manager.cc:1145] Deleting tablet 210f161cf6b94a2abbad651c40bb8cec
0115 14:52:10.579884 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.579950 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373568}
W20250115 14:52:10.580487 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 210f161cf6b94a2abbad651c40bb8cec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 210f161cf6b94a2abbad651c40bb8cec already in progress: creating tablet
I20250115 14:52:10.581149 1014 tablet_service.cc:1467] Processing CreateTablet for tablet a3b82825622b45c7aa1f076e83c1ae78 (DEFAULT_TABLE table=default.Table_1b68293cce4a49c3b939caef23d5e36c [id=588269769c5f4f5da4d677e56daf1363]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.581518 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3b82825622b45c7aa1f076e83c1ae78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.581485 753 tablet_service.cc:1467] Processing CreateTablet for tablet b333d2c5a53745b9aeca2726318f628c (DEFAULT_TABLE table=default.Table_3991253a712b4579981713c7f964a053 [id=605fa34a0801465da125071f7cb88fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.581490 880 tablet_service.cc:1467] Processing CreateTablet for tablet b333d2c5a53745b9aeca2726318f628c (DEFAULT_TABLE table=default.Table_3991253a712b4579981713c7f964a053 [id=605fa34a0801465da125071f7cb88fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.581822 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b333d2c5a53745b9aeca2726318f628c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.581823 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b333d2c5a53745b9aeca2726318f628c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.589358 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b333d2c5a53745b9aeca2726318f628c (DEFAULT_TABLE table=default.Table_3991253a712b4579981713c7f964a053 [id=605fa34a0801465da125071f7cb88fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.589699 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b333d2c5a53745b9aeca2726318f628c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.589859 1168 tablet_bootstrap.cc:492] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:10.589979 1168 ts_tablet_manager.cc:1397] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.003s
I20250115 14:52:10.590741 1168 raft_consensus.cc:357] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.590977 1168 raft_consensus.cc:383] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.591029 1168 raft_consensus.cc:738] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.591199 1168 consensus_queue.cc:260] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.591672 1168 ts_tablet_manager.cc:1428] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:10.591920 1168 tablet_bootstrap.cc:492] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.592904 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 900) took 2337 ms. Trace:
I20250115 14:52:10.592978 1167 rpcz_store.cc:276] 0115 14:52:08.255063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.255312 (+ 249us) service_pool.cc:225] Handling call
0115 14:52:10.580232 (+2324920us) ts_tablet_manager.cc:1145] Deleting tablet e5dfc00ecec54f92a75fa77c84fe8a6d
0115 14:52:10.580239 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.592884 (+ 12645us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2324734}
W20250115 14:52:10.593389 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e5dfc00ecec54f92a75fa77c84fe8a6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5dfc00ecec54f92a75fa77c84fe8a6d already in progress: creating tablet
I20250115 14:52:10.596558 755 tablet_service.cc:1467] Processing CreateTablet for tablet d40be9c7d719498896a93c07a549280c (DEFAULT_TABLE table=default.Table_0be69f26c9784abb83eee94fe5bfdc39 [id=d2d9f3f89f2c4422afab3f1819940b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.596916 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d40be9c7d719498896a93c07a549280c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.597154 1166 ts_tablet_manager.cc:1905] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.602589 1166 ts_tablet_manager.cc:1918] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:10.602675 1166 log.cc:1198] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/8442dc919d5f4d63b9b2ce7276cc0bc5
I20250115 14:52:10.602795 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6a991cf39a43ceb649c8fcb54e837c (DEFAULT_TABLE table=default.Table_ae6d939a2d5e4be482722ce8098cb5d8 [id=a0d48729ea2c493baf1515387143ecc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.603107 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6a991cf39a43ceb649c8fcb54e837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.603106 1166 ts_tablet_manager.cc:1939] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.603622 876 tablet_service.cc:1467] Processing CreateTablet for tablet d40be9c7d719498896a93c07a549280c (DEFAULT_TABLE table=default.Table_0be69f26c9784abb83eee94fe5bfdc39 [id=d2d9f3f89f2c4422afab3f1819940b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.603960 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d40be9c7d719498896a93c07a549280c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.605855 882 tablet_service.cc:1467] Processing CreateTablet for tablet fc741c3e00b543f99cf68122f58fd111 (DEFAULT_TABLE table=default.Table_3ac29fd7d7614bd98b6ab37861ad016e [id=505b22e3ae0b421c9398dd418678ede2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.606154 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc741c3e00b543f99cf68122f58fd111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.606858 875 tablet_service.cc:1467] Processing CreateTablet for tablet bfc6e5c81a6d40a3a2054450d0478ad6 (DEFAULT_TABLE table=default.Table_132293624dd74f678fbc0f2204143d28 [id=09bdc6f645284c71a1f3903f11221052]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.607158 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfc6e5c81a6d40a3a2054450d0478ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.608523 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc3e6bd0aee457186cdffcd0f5bff77 (DEFAULT_TABLE table=default.Table_64bd6866f7624f7eaecc8e0486d14b21 [id=e373314796b24eca8895e4678471e473]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.608841 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc3e6bd0aee457186cdffcd0f5bff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.612936 752 tablet_service.cc:1467] Processing CreateTablet for tablet fc741c3e00b543f99cf68122f58fd111 (DEFAULT_TABLE table=default.Table_3ac29fd7d7614bd98b6ab37861ad016e [id=505b22e3ae0b421c9398dd418678ede2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.613286 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc741c3e00b543f99cf68122f58fd111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.616539 748 tablet_service.cc:1467] Processing CreateTablet for tablet bfc6e5c81a6d40a3a2054450d0478ad6 (DEFAULT_TABLE table=default.Table_132293624dd74f678fbc0f2204143d28 [id=09bdc6f645284c71a1f3903f11221052]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.616845 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfc6e5c81a6d40a3a2054450d0478ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.616847 1166 tablet_replica.cc:331] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.617539 1166 raft_consensus.cc:2238] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.617668 1166 raft_consensus.cc:2267] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.618409 1166 ts_tablet_manager.cc:1905] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.621157 2273 raft_consensus.cc:491] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.621282 2273 raft_consensus.cc:513] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.621855 2273 leader_election.cc:290] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.622787 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f0ecfceecf4acdbf2c2fbec88ff3cb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:10.624418 984 leader_election.cc:343] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.629390 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 (table default.Table_b3a6b57353484cdf8f2dfde2738ae0c2 [id=5249bd4dd6f14f9bbfced76f1d336c71]) successfully deleted
W20250115 14:52:10.629678 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet efb8b001dd66456d9e49e9e8a6848ccf because the tablet was not found. No further retry: Not found: Tablet not found: efb8b001dd66456d9e49e9e8a6848ccf
W20250115 14:52:10.631379 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b28a180aa0e4ef2b62890233cd512a4 consensus state has no leader
W20250115 14:52:10.631994 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92aeca346a7746468dbbff053d933472 consensus state has no leader
I20250115 14:52:10.633083 1015 tablet_service.cc:1467] Processing CreateTablet for tablet fc741c3e00b543f99cf68122f58fd111 (DEFAULT_TABLE table=default.Table_3ac29fd7d7614bd98b6ab37861ad016e [id=505b22e3ae0b421c9398dd418678ede2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.633464 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc741c3e00b543f99cf68122f58fd111. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.633891 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 64a76d2772534687b1391e45b9cb984b
I20250115 14:52:10.634903 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 13053264b5ca41dd85e53d928aa511fd
I20250115 14:52:10.635391 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3963e79868284401a915cac5ef258155
I20250115 14:52:10.637691 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 3963e79868284401a915cac5ef258155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.637898 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a76d2772534687b1391e45b9cb984b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.637899 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3963e79868284401a915cac5ef258155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.638070 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 13053264b5ca41dd85e53d928aa511fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.638082 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 13053264b5ca41dd85e53d928aa511fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.638422 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc3e6bd0aee457186cdffcd0f5bff77 (DEFAULT_TABLE table=default.Table_64bd6866f7624f7eaecc8e0486d14b21 [id=e373314796b24eca8895e4678471e473]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.638769 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc3e6bd0aee457186cdffcd0f5bff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.639498 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d40be9c7d719498896a93c07a549280c (DEFAULT_TABLE table=default.Table_0be69f26c9784abb83eee94fe5bfdc39 [id=d2d9f3f89f2c4422afab3f1819940b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.639843 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d40be9c7d719498896a93c07a549280c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.640707 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a76d2772534687b1391e45b9cb984b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.643522 1166 ts_tablet_manager.cc:1918] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.643612 1166 log.cc:1198] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9e95d09efd2e44cc81cdcbf87ee453fe
I20250115 14:52:10.643872 1166 ts_tablet_manager.cc:1939] T 9e95d09efd2e44cc81cdcbf87ee453fe P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.644870 1164 tablet_bootstrap.cc:492] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.644979 1164 ts_tablet_manager.cc:1397] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.121s user 0.000s sys 0.002s
I20250115 14:52:10.645229 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc3e6bd0aee457186cdffcd0f5bff77 (DEFAULT_TABLE table=default.Table_64bd6866f7624f7eaecc8e0486d14b21 [id=e373314796b24eca8895e4678471e473]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.645560 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc3e6bd0aee457186cdffcd0f5bff77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.645691 1164 raft_consensus.cc:357] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.645865 1164 raft_consensus.cc:383] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.645905 1164 raft_consensus.cc:738] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.646116 1164 consensus_queue.cc:260] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.646587 1164 ts_tablet_manager.cc:1428] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:10.646842 1164 tablet_bootstrap.cc:492] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.648561 880 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6a991cf39a43ceb649c8fcb54e837c (DEFAULT_TABLE table=default.Table_ae6d939a2d5e4be482722ce8098cb5d8 [id=a0d48729ea2c493baf1515387143ecc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.648643 2340 raft_consensus.cc:491] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.648924 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6a991cf39a43ceb649c8fcb54e837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.648823 2340 raft_consensus.cc:513] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.649251 2480 tablet.cc:1721] T df84bc24a71e431cb47b93f2b0dcaa90 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:10.649407 2340 leader_election.cc:290] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:10.650270 848 leader_election.cc:343] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Not found: Tablet not found: 5b85d63974ab418eb1921dd605fbf2b0
I20250115 14:52:10.652534 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 901) took 2331 ms. Trace:
I20250115 14:52:10.652609 1167 rpcz_store.cc:276] 0115 14:52:08.321540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.321598 (+ 58us) service_pool.cc:225] Handling call
0115 14:52:10.652424 (+2330826us) ts_tablet_manager.cc:1145] Deleting tablet 929830d8e8fd4b0c8a7b18c941706929
0115 14:52:10.652433 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.652517 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2330604}
W20250115 14:52:10.653162 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 929830d8e8fd4b0c8a7b18c941706929 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 929830d8e8fd4b0c8a7b18c941706929 already in progress: creating tablet
I20250115 14:52:10.653564 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 902) took 2320 ms. Trace:
I20250115 14:52:10.653646 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f0ecfceecf4acdbf2c2fbec88ff3cb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:10.654162 983 leader_election.cc:343] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.654260 983 leader_election.cc:304] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.654582 2273 raft_consensus.cc:2743] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.655126 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 248cadcdef8b4e99923cdbca39690685 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.655395 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 3963e79868284401a915cac5ef258155 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.655421 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 64a76d2772534687b1391e45b9cb984b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.655599 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 5265d5eeac83481ca97801e3475886da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.655718 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 1140016d11a74a2cadba494c0b319c54 with delete_type TABLET_DATA_DELETED (replaced by tablet 068771fee92f403d8997646291a83dc6) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.655836 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 13053264b5ca41dd85e53d928aa511fd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.656050 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9727430a15574ec0b24a5a64da744cac
W20250115 14:52:10.656853 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44c70b9eba794f6a8ee393679db01a4c has no consensus state
I20250115 14:52:10.657096 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 9727430a15574ec0b24a5a64da744cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.657259 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c6f9262e808b4589a21257ed3fb0363b
I20250115 14:52:10.657706 455 catalog_manager.cc:6596] ReplaceTablet: tablet c6f9262e808b4589a21257ed3fb0363b deleted and replaced by tablet 17aa6950326d4226976e07c7cc986d39
I20250115 14:52:10.658120 875 tablet_service.cc:1514] Processing DeleteTablet for tablet c6f9262e808b4589a21257ed3fb0363b with delete_type TABLET_DATA_DELETED (replaced by tablet 17aa6950326d4226976e07c7cc986d39) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.659139 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet c6f9262e808b4589a21257ed3fb0363b with delete_type TABLET_DATA_DELETED (replaced by tablet 17aa6950326d4226976e07c7cc986d39) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.659160 451 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet df84bc24a71e431cb47b93f2b0dcaa90 (table default.Table_5ff3a565903a49c4b0d981883ca6f331 [id=c0c9df829b7c4202b8c36d7d5d767168]). Expected version 1 got 0
I20250115 14:52:10.659389 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 9727430a15574ec0b24a5a64da744cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.659409 451 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 2484885df0da46f381d625a26e211c1e (table default.Table_8652d6d9ec084553b5183aa372c2554a [id=c29c4105b83d4df9b617997c0c9c5b14]). Expected version 1 got 0
I20250115 14:52:10.662087 748 tablet_service.cc:1514] Processing DeleteTablet for tablet c6f9262e808b4589a21257ed3fb0363b with delete_type TABLET_DATA_DELETED (replaced by tablet 17aa6950326d4226976e07c7cc986d39) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.662485 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 9727430a15574ec0b24a5a64da744cac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.664233 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9b6a991cf39a43ceb649c8fcb54e837c (DEFAULT_TABLE table=default.Table_ae6d939a2d5e4be482722ce8098cb5d8 [id=a0d48729ea2c493baf1515387143ecc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.664705 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6a991cf39a43ceb649c8fcb54e837c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.667275 2273 raft_consensus.cc:491] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.667402 2273 raft_consensus.cc:513] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.667977 2273 leader_election.cc:290] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.668746 1164 tablet_bootstrap.cc:654] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.668907 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7e70288eb248f1a71646c66e376a63" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:10.669478 983 leader_election.cc:343] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.653640 1167 rpcz_store.cc:276] 0115 14:52:08.333304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.333944 (+ 640us) service_pool.cc:225] Handling call
0115 14:52:10.653462 (+2319518us) ts_tablet_manager.cc:1145] Deleting tablet a349cfaff6be4b85a0b2cf3eee238c26
0115 14:52:10.653471 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.653548 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2319346}
W20250115 14:52:10.670037 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a349cfaff6be4b85a0b2cf3eee238c26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a349cfaff6be4b85a0b2cf3eee238c26 already in progress: creating tablet
I20250115 14:52:10.670425 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 903) took 2258 ms. Trace:
I20250115 14:52:10.670496 1167 rpcz_store.cc:276] 0115 14:52:08.411488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.411545 (+ 57us) service_pool.cc:225] Handling call
0115 14:52:10.670313 (+2258768us) ts_tablet_manager.cc:1145] Deleting tablet 1140016d11a74a2cadba494c0b319c54
0115 14:52:10.670324 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.670409 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2258585}
W20250115 14:52:10.670859 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 1140016d11a74a2cadba494c0b319c54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1140016d11a74a2cadba494c0b319c54 already in progress: creating tablet
I20250115 14:52:10.671195 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 907) took 2236 ms. Trace:
I20250115 14:52:10.671262 1167 rpcz_store.cc:276] 0115 14:52:08.435159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.435736 (+ 577us) service_pool.cc:225] Handling call
0115 14:52:10.671096 (+2235360us) ts_tablet_manager.cc:1145] Deleting tablet 221b3271c2734828a96c4c86ab031dfc
0115 14:52:10.671106 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.671179 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2235202}
W20250115 14:52:10.671633 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 221b3271c2734828a96c4c86ab031dfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 221b3271c2734828a96c4c86ab031dfc already in progress: creating tablet
I20250115 14:52:10.671952 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 908) took 2236 ms. Trace:
I20250115 14:52:10.672032 1167 rpcz_store.cc:276] 0115 14:52:08.435254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.436704 (+ 1450us) service_pool.cc:225] Handling call
0115 14:52:10.671848 (+2235144us) ts_tablet_manager.cc:1145] Deleting tablet 7654d2831f0141858b52379b767b3a70
0115 14:52:10.671858 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.671934 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2235009}
W20250115 14:52:10.672405 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7654d2831f0141858b52379b767b3a70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7654d2831f0141858b52379b767b3a70 already in progress: creating tablet
I20250115 14:52:10.672709 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 910) took 2199 ms. Trace:
I20250115 14:52:10.672773 1167 rpcz_store.cc:276] 0115 14:52:08.472829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.473190 (+ 361us) service_pool.cc:225] Handling call
0115 14:52:10.672605 (+2199415us) ts_tablet_manager.cc:1145] Deleting tablet f400880835ab4efdb2af0be5553d20dc
0115 14:52:10.672614 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.672693 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2199269}
W20250115 14:52:10.673152 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f400880835ab4efdb2af0be5553d20dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f400880835ab4efdb2af0be5553d20dc already in progress: creating tablet
I20250115 14:52:10.673393 1167 tablet_replica.cc:331] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:10.673563 1167 raft_consensus.cc:2238] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:10.673827 1167 pending_rounds.cc:62] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:10.673866 1167 pending_rounds.cc:69] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558375831773184 op_type: NO_OP noop_request { }
I20250115 14:52:10.673949 1167 raft_consensus.cc:2883] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:10.674021 1167 raft_consensus.cc:2267] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.674587 1167 ts_tablet_manager.cc:1905] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.680222 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_5ff3a565903a49c4b0d981883ca6f331 [id=c0c9df829b7c4202b8c36d7d5d767168] alter complete (version 1)
I20250115 14:52:10.682662 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 6942d102069d4c33b23fa2c0b29f94a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.683143 1166 tablet_replica.cc:331] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.684465 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b85d63974ab418eb1921dd605fbf2b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.685205 752 tablet_service.cc:1514] Processing DeleteTablet for tablet a600fa0f0ec049dca2803bf7d2c74a95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.685814 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8e2ac157ed9045618562bb7d16dd9aaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.686381 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e2ac157ed9045618562bb7d16dd9aaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.687217 1168 tablet_bootstrap.cc:654] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.687570 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_23055a20564d4db3b53d4876e1995d07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.685829 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9e95d09efd2e44cc81cdcbf87ee453fe (table default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]) successfully deleted
I20250115 14:52:10.688278 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_672b45974cb345618b2af49aaed2365e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.688778 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_672b45974cb345618b2af49aaed2365e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.688068 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23055a20564d4db3b53d4876e1995d07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.689908 875 tablet_service.cc:1467] Processing CreateTablet for tablet 41cf85429fd140519f81141d21bf58b6 (DEFAULT_TABLE table=default.Table_303f3f6394c54dd2b5a622b59dee47cb [id=619ecb178dc14e0a8fc03896c9a4cdc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.691157 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7e70288eb248f1a71646c66e376a63" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:10.691619 876 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf17435b88c4e87800daefcd4fa35ec (DEFAULT_TABLE table=default.Table_b8c3f1232b1a40d0a1110de0eb3289a6 [id=515cd08576c84073a01fb61d7f0bb5a2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:10.691684 984 leader_election.cc:343] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.691809 984 leader_election.cc:304] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.692162 2273 raft_consensus.cc:2743] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.692516 893 tablet_service.cc:1467] Processing CreateTablet for tablet 674176b447064f33a21fcfcb99e7fac6 (DEFAULT_TABLE table=default.Table_2d510cf7315e46bab6863551ac6bf816 [id=2ca3d3511d1b45deaa2e1762bb45d044]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.692683 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf17435b88c4e87800daefcd4fa35ec (DEFAULT_TABLE table=default.Table_b8c3f1232b1a40d0a1110de0eb3289a6 [id=515cd08576c84073a01fb61d7f0bb5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.693035 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf17435b88c4e87800daefcd4fa35ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.693337 882 tablet_service.cc:1467] Processing CreateTablet for tablet 412100cf34eb4d40988dca4c4bea03f3 (DEFAULT_TABLE table=default.Table_be9185f83999438bae9ef0d3dca88fd7 [id=4bf91c4f247c4c2a93b4aa9cc658edaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.693686 752 tablet_service.cc:1467] Processing CreateTablet for tablet 41cf85429fd140519f81141d21bf58b6 (DEFAULT_TABLE table=default.Table_303f3f6394c54dd2b5a622b59dee47cb [id=619ecb178dc14e0a8fc03896c9a4cdc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.693928 751 tablet_service.cc:1467] Processing CreateTablet for tablet 674176b447064f33a21fcfcb99e7fac6 (DEFAULT_TABLE table=default.Table_2d510cf7315e46bab6863551ac6bf816 [id=2ca3d3511d1b45deaa2e1762bb45d044]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.694032 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41cf85429fd140519f81141d21bf58b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.694239 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 674176b447064f33a21fcfcb99e7fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.694461 748 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf17435b88c4e87800daefcd4fa35ec (DEFAULT_TABLE table=default.Table_b8c3f1232b1a40d0a1110de0eb3289a6 [id=515cd08576c84073a01fb61d7f0bb5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.694819 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf17435b88c4e87800daefcd4fa35ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.696516 740 tablet_service.cc:1467] Processing CreateTablet for tablet 412100cf34eb4d40988dca4c4bea03f3 (DEFAULT_TABLE table=default.Table_be9185f83999438bae9ef0d3dca88fd7 [id=4bf91c4f247c4c2a93b4aa9cc658edaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.696836 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412100cf34eb4d40988dca4c4bea03f3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:10.702804 847 leader_election.cc:343] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 5b85d63974ab418eb1921dd605fbf2b0
I20250115 14:52:10.702940 847 leader_election.cc:304] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.703625 2341 raft_consensus.cc:2743] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.708328 1015 tablet_service.cc:1467] Processing CreateTablet for tablet bfc6e5c81a6d40a3a2054450d0478ad6 (DEFAULT_TABLE table=default.Table_132293624dd74f678fbc0f2204143d28 [id=09bdc6f645284c71a1f3903f11221052]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.685257 1166 raft_consensus.cc:2238] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.708762 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfc6e5c81a6d40a3a2054450d0478ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.709132 1166 raft_consensus.cc:2267] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.709950 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 674176b447064f33a21fcfcb99e7fac6 (DEFAULT_TABLE table=default.Table_2d510cf7315e46bab6863551ac6bf816 [id=2ca3d3511d1b45deaa2e1762bb45d044]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.710135 1166 ts_tablet_manager.cc:1905] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.710333 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 674176b447064f33a21fcfcb99e7fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.710768 1164 tablet_bootstrap.cc:492] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.710937 1164 ts_tablet_manager.cc:1397] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.064s user 0.000s sys 0.003s
I20250115 14:52:10.711650 1164 raft_consensus.cc:357] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.711895 1164 raft_consensus.cc:383] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.711989 1164 raft_consensus.cc:738] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.712234 1164 consensus_queue.cc:260] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.712759 1164 ts_tablet_manager.cc:1428] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:10.713052 1164 tablet_bootstrap.cc:492] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.714721 1164 tablet_bootstrap.cc:654] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.715610 1166 ts_tablet_manager.cc:1918] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.715708 1166 log.cc:1198] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/282d988bc41443c381bb0ba69b618f5d
I20250115 14:52:10.715966 1166 ts_tablet_manager.cc:1939] T 282d988bc41443c381bb0ba69b618f5d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.716675 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6c9c88e11c6d4580889dd2b45811a57f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.717265 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c9c88e11c6d4580889dd2b45811a57f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.718091 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d447472ba817465d94a1a16085f8d1cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.718592 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d447472ba817465d94a1a16085f8d1cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.719907 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 41cf85429fd140519f81141d21bf58b6 (DEFAULT_TABLE table=default.Table_303f3f6394c54dd2b5a622b59dee47cb [id=619ecb178dc14e0a8fc03896c9a4cdc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.720325 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41cf85429fd140519f81141d21bf58b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.720607 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a0baa54d6f6f437d954be2ba0f6c03d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.721048 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0baa54d6f6f437d954be2ba0f6c03d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.721164 2273 raft_consensus.cc:491] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.721273 2273 raft_consensus.cc:513] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.721827 2273 leader_election.cc:290] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.722626 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df0250d08214a9783c392071126ed4a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:10.723124 983 leader_election.cc:343] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.728428 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a000ab73dd46ac8e134784a156040a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.729326 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf17435b88c4e87800daefcd4fa35ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.729534 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412100cf34eb4d40988dca4c4bea03f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.729777 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 674176b447064f33a21fcfcb99e7fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.729982 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41cf85429fd140519f81141d21bf58b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.738076 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_220ee236fc72425ab7480c72858f2989"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.738667 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_220ee236fc72425ab7480c72858f2989 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.740729 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2b06a00240dc4c3c9c3c257319305a60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.741361 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b06a00240dc4c3c9c3c257319305a60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.741631 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c3e4c1d6466242fb8b2b5d97d504836b" } new_table_name: "default.Table_0d800cc4760a43fc82ef693a68ef4928" modify_external_catalogs: true
I20250115 14:52:10.743813 1162 tablet_bootstrap.cc:492] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.744016 1162 ts_tablet_manager.cc:1397] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.316s user 0.006s sys 0.001s
I20250115 14:52:10.745334 1162 raft_consensus.cc:357] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.745570 1162 raft_consensus.cc:383] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.745653 1162 raft_consensus.cc:738] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.745915 1162 consensus_queue.cc:260] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.746675 1162 ts_tablet_manager.cc:1428] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:52:10.746963 2273 raft_consensus.cc:491] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.747087 2273 raft_consensus.cc:513] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.747615 2273 leader_election.cc:290] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.752389 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e616d78933e4f84b9ec2acc51450b18" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:10.752965 983 leader_election.cc:343] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.756731 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 412100cf34eb4d40988dca4c4bea03f3 (DEFAULT_TABLE table=default.Table_be9185f83999438bae9ef0d3dca88fd7 [id=4bf91c4f247c4c2a93b4aa9cc658edaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.757805 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8e0ef20f9940b1ae1008402052ccdf (DEFAULT_TABLE table=default.Table_d6b3a467e1e2479b9734b57db305df67 [id=d971549143e9480b9e92cd3abe61f334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.758472 1024 tablet_service.cc:1467] Processing CreateTablet for tablet c5ab056252b5435693e0d5c041c9e1dc (DEFAULT_TABLE table=default.Table_ab72fac2354c4767b21a84c1a2fc5dec [id=9eb93eb7a4334342995e977a4aae5c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.759143 1019 tablet_service.cc:1467] Processing CreateTablet for tablet b56ad82b94964cacb5fd781b25ff0d83 (DEFAULT_TABLE table=default.Table_6f324c73cb6341b68a7b0d3947954ae0 [id=3cad4c69043f4ee3b5e67b4b56e77d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.759987 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 42d8c79e0f7a444c869378678344f2b7 (DEFAULT_TABLE table=default.Table_8bab10f63d3c43eca6cda9f4c2d4189a [id=9e41a8397db94127a19c471caaae75e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.760721 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 5b33c876fab1476aba3b1f6e799fa96c (DEFAULT_TABLE table=default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 [id=cdf45f410fda46f29cd6dbfd1cc45f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.762202 740 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8e0ef20f9940b1ae1008402052ccdf (DEFAULT_TABLE table=default.Table_d6b3a467e1e2479b9734b57db305df67 [id=d971549143e9480b9e92cd3abe61f334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.762732 882 tablet_service.cc:1467] Processing CreateTablet for tablet b56ad82b94964cacb5fd781b25ff0d83 (DEFAULT_TABLE table=default.Table_6f324c73cb6341b68a7b0d3947954ae0 [id=3cad4c69043f4ee3b5e67b4b56e77d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.762863 876 tablet_service.cc:1467] Processing CreateTablet for tablet c5ab056252b5435693e0d5c041c9e1dc (DEFAULT_TABLE table=default.Table_ab72fac2354c4767b21a84c1a2fc5dec [id=9eb93eb7a4334342995e977a4aae5c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.762848 751 tablet_service.cc:1467] Processing CreateTablet for tablet b56ad82b94964cacb5fd781b25ff0d83 (DEFAULT_TABLE table=default.Table_6f324c73cb6341b68a7b0d3947954ae0 [id=3cad4c69043f4ee3b5e67b4b56e77d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.763125 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56ad82b94964cacb5fd781b25ff0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.763192 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56ad82b94964cacb5fd781b25ff0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.763562 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e616d78933e4f84b9ec2acc51450b18" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:10.764081 984 leader_election.cc:343] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.764183 984 leader_election.cc:304] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.765008 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5ab056252b5435693e0d5c041c9e1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.767805 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c8e0ef20f9940b1ae1008402052ccdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.768415 2273 raft_consensus.cc:2743] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.768940 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df0250d08214a9783c392071126ed4a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:10.769469 984 leader_election.cc:343] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.769572 984 leader_election.cc:304] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.769654 755 tablet_service.cc:1467] Processing CreateTablet for tablet c5ab056252b5435693e0d5c041c9e1dc (DEFAULT_TABLE table=default.Table_ab72fac2354c4767b21a84c1a2fc5dec [id=9eb93eb7a4334342995e977a4aae5c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.769950 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5ab056252b5435693e0d5c041c9e1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.769980 2273 raft_consensus.cc:2743] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.770401 875 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8e0ef20f9940b1ae1008402052ccdf (DEFAULT_TABLE table=default.Table_d6b3a467e1e2479b9734b57db305df67 [id=d971549143e9480b9e92cd3abe61f334]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.770923 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c8e0ef20f9940b1ae1008402052ccdf. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:10.775677 848 consensus_peers.cc:487] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3b75d759f834d558c94cc981e338ac0. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:10.777626 2273 raft_consensus.cc:491] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.777741 882 tablet_service.cc:1467] Processing CreateTablet for tablet 5b33c876fab1476aba3b1f6e799fa96c (DEFAULT_TABLE table=default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 [id=cdf45f410fda46f29cd6dbfd1cc45f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.777745 2273 raft_consensus.cc:513] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.778050 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b33c876fab1476aba3b1f6e799fa96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.778339 2273 leader_election.cc:290] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.778995 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1140016d11a74a2cadba494c0b319c54" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:10.779239 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1140016d11a74a2cadba494c0b319c54" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:10.780364 984 leader_election.cc:343] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:10.780601 983 leader_election.cc:343] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.780676 983 leader_election.cc:304] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.781037 2273 raft_consensus.cc:2743] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.783093 748 tablet_service.cc:1467] Processing CreateTablet for tablet 42d8c79e0f7a444c869378678344f2b7 (DEFAULT_TABLE table=default.Table_8bab10f63d3c43eca6cda9f4c2d4189a [id=9e41a8397db94127a19c471caaae75e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.783690 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1d252b88a14246be9189c5c5261887c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.784266 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d252b88a14246be9189c5c5261887c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.785053 1164 tablet_bootstrap.cc:492] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.785166 1164 ts_tablet_manager.cc:1397] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.000s
I20250115 14:52:10.785501 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_13267a7581594c869ec943c3a7f1058e" } modify_external_catalogs: true
I20250115 14:52:10.785657 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_13267a7581594c869ec943c3a7f1058e" } modify_external_catalogs: true
I20250115 14:52:10.785934 1164 raft_consensus.cc:357] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.786116 1164 raft_consensus.cc:383] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.786162 1164 raft_consensus.cc:738] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.786315 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b33c876fab1476aba3b1f6e799fa96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.786780 1168 tablet_bootstrap.cc:492] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:10.786895 1168 ts_tablet_manager.cc:1397] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.195s user 0.003s sys 0.000s
I20250115 14:52:10.786875 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_67aaf987cc30461bb07c319c25639ffa" } new_table_name: "default.Table_c1296b8b98f242948cfbcea7f0158ee5" modify_external_catalogs: true
I20250115 14:52:10.787554 1168 raft_consensus.cc:357] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.787717 1168 raft_consensus.cc:383] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.787768 1168 raft_consensus.cc:738] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.787943 1168 consensus_queue.cc:260] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.788779 1168 ts_tablet_manager.cc:1428] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:10.789042 1168 tablet_bootstrap.cc:492] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.790295 1168 tablet_bootstrap.cc:654] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.794032 1167 ts_tablet_manager.cc:1918] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:10.794117 1167 log.cc:1198] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/43605c995df4400e91139199c9514bfd
I20250115 14:52:10.794260 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 412100cf34eb4d40988dca4c4bea03f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.794409 1167 ts_tablet_manager.cc:1939] T 43605c995df4400e91139199c9514bfd P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:10.797336 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d8c79e0f7a444c869378678344f2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.799896 1164 consensus_queue.cc:260] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.800530 1164 ts_tablet_manager.cc:1428] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250115 14:52:10.800858 1164 tablet_bootstrap.cc:492] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.802630 1164 tablet_bootstrap.cc:654] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.803905 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 742792be08444792892bd9cd7336dda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.805851 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 638507d08f654011bea2597c22a788a2 (DEFAULT_TABLE table=default.Table_af7ebd20849b40729c5650194d55240f [id=092f000963cd4200a9e97e353b98c74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.806197 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 638507d08f654011bea2597c22a788a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.806185 893 tablet_service.cc:1467] Processing CreateTablet for tablet 651ea2c7d1d7447a9ac69b1b120f19cd (DEFAULT_TABLE table=default.Table_7bc6f9ea70464e1cb86a77b5fb2f48dc [id=8d43f22bf3e14cc6b993f29753b5cb28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.806553 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 651ea2c7d1d7447a9ac69b1b120f19cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.806855 882 tablet_service.cc:1467] Processing CreateTablet for tablet 638507d08f654011bea2597c22a788a2 (DEFAULT_TABLE table=default.Table_af7ebd20849b40729c5650194d55240f [id=092f000963cd4200a9e97e353b98c74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.807153 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 638507d08f654011bea2597c22a788a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.806191 880 tablet_service.cc:1467] Processing CreateTablet for tablet ef1aa82585764f75807bd3de3d2a5caf (DEFAULT_TABLE table=default.Table_ead9eb0fd5884f6abacc69ebb3213e77 [id=73dfdc357a69414db39cce3049d38e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.807472 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1aa82585764f75807bd3de3d2a5caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.809537 737 tablet_service.cc:1467] Processing CreateTablet for tablet ef1aa82585764f75807bd3de3d2a5caf (DEFAULT_TABLE table=default.Table_ead9eb0fd5884f6abacc69ebb3213e77 [id=73dfdc357a69414db39cce3049d38e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.809878 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1aa82585764f75807bd3de3d2a5caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.812530 752 tablet_service.cc:1467] Processing CreateTablet for tablet 638507d08f654011bea2597c22a788a2 (DEFAULT_TABLE table=default.Table_af7ebd20849b40729c5650194d55240f [id=092f000963cd4200a9e97e353b98c74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.812880 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 638507d08f654011bea2597c22a788a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.814730 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 651ea2c7d1d7447a9ac69b1b120f19cd (DEFAULT_TABLE table=default.Table_7bc6f9ea70464e1cb86a77b5fb2f48dc [id=8d43f22bf3e14cc6b993f29753b5cb28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.815131 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 651ea2c7d1d7447a9ac69b1b120f19cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.815120 740 tablet_service.cc:1467] Processing CreateTablet for tablet 5b33c876fab1476aba3b1f6e799fa96c (DEFAULT_TABLE table=default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 [id=cdf45f410fda46f29cd6dbfd1cc45f28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.815794 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b33c876fab1476aba3b1f6e799fa96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.824923 753 tablet_service.cc:1467] Processing CreateTablet for tablet 651ea2c7d1d7447a9ac69b1b120f19cd (DEFAULT_TABLE table=default.Table_7bc6f9ea70464e1cb86a77b5fb2f48dc [id=8d43f22bf3e14cc6b993f29753b5cb28]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.825285 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 651ea2c7d1d7447a9ac69b1b120f19cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.826268 1162 tablet_bootstrap.cc:492] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:10.827620 1162 tablet_bootstrap.cc:654] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.831394 1166 tablet_replica.cc:331] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:10.831844 1166 raft_consensus.cc:2238] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.832355 1166 raft_consensus.cc:2267] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.838042 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d8c79e0f7a444c869378678344f2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.838634 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9e128687957b4b559bb77fd82039e563" } modify_external_catalogs: true
I20250115 14:52:10.838850 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9e128687957b4b559bb77fd82039e563" } modify_external_catalogs: true
I20250115 14:52:10.839359 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 282d988bc41443c381bb0ba69b618f5d (table default.Table_4ab29505c6634887aa99118862eccf5d [id=83320921ba86467b9ad31fde698cb2d1]) successfully deleted
I20250115 14:52:10.840849 882 tablet_service.cc:1467] Processing CreateTablet for tablet 0936fa510a454b8e91d6ef2b345b18df (DEFAULT_TABLE table=default.Table_0610ab6883db4caa82a59fbf030bfedf [id=4c4b512016a24d24b6788075aeb0c3cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.841406 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0936fa510a454b8e91d6ef2b345b18df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.842226 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d447472ba817465d94a1a16085f8d1cf" } modify_external_catalogs: true
I20250115 14:52:10.842388 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d447472ba817465d94a1a16085f8d1cf" } modify_external_catalogs: true
I20250115 14:52:10.843497 879 tablet_service.cc:1467] Processing CreateTablet for tablet 1f354e4204ec47149957eb22f964f08a (DEFAULT_TABLE table=default.Table_f257047f59814e1a8458779252a782b9 [id=51d11f13250a43e8bc3b6988128d76d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.843817 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f354e4204ec47149957eb22f964f08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.844012 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b56ad82b94964cacb5fd781b25ff0d83. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.845971 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 911) took 2373 ms. Trace:
I20250115 14:52:10.846071 1167 rpcz_store.cc:276] 0115 14:52:08.472938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.473365 (+ 427us) service_pool.cc:225] Handling call
0115 14:52:10.673354 (+2199989us) ts_tablet_manager.cc:1145] Deleting tablet 43605c995df4400e91139199c9514bfd
0115 14:52:10.673362 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.750562 (+ 77200us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.793940 (+ 43378us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.845952 (+ 52012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2199887}
I20250115 14:52:10.846470 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 43605c995df4400e91139199c9514bfd (table default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]) successfully deleted
I20250115 14:52:10.846736 1167 tablet_replica.cc:331] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:10.846916 1167 raft_consensus.cc:2238] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.847051 1167 raft_consensus.cc:2267] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.848011 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5ab056252b5435693e0d5c041c9e1dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.851756 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5f0eeede8f714b92b1e3a9a558289c3e
I20250115 14:52:10.852460 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c8e0ef20f9940b1ae1008402052ccdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.855975 1168 tablet_bootstrap.cc:492] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:10.856115 1168 ts_tablet_manager.cc:1397] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.001s
I20250115 14:52:10.856786 1168 raft_consensus.cc:357] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.856977 1168 raft_consensus.cc:383] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.857039 1168 raft_consensus.cc:738] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.857250 1168 consensus_queue.cc:260] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.857707 1168 ts_tablet_manager.cc:1428] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:10.857945 1168 tablet_bootstrap.cc:492] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.858179 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0eeede8f714b92b1e3a9a558289c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.859196 1168 tablet_bootstrap.cc:654] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.860652 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_65167815994b4ed4aa3d30cb8b168d63" } new_table_name: "default.Table_7a882c6c77fd4d198c18380628c8ef1a" modify_external_catalogs: true
I20250115 14:52:10.861770 1010 tablet_service.cc:1467] Processing CreateTablet for tablet ef1aa82585764f75807bd3de3d2a5caf (DEFAULT_TABLE table=default.Table_ead9eb0fd5884f6abacc69ebb3213e77 [id=73dfdc357a69414db39cce3049d38e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.862418 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_672b45974cb345618b2af49aaed2365e" } new_table_name: "default.Table_c49292ffc41542abb7c1104187ce5cba" modify_external_catalogs: true
I20250115 14:52:10.862511 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef1aa82585764f75807bd3de3d2a5caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.863370 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 1f354e4204ec47149957eb22f964f08a (DEFAULT_TABLE table=default.Table_f257047f59814e1a8458779252a782b9 [id=51d11f13250a43e8bc3b6988128d76d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.864504 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8bab10f63d3c43eca6cda9f4c2d4189a" } modify_external_catalogs: true
I20250115 14:52:10.864665 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8bab10f63d3c43eca6cda9f4c2d4189a" } modify_external_catalogs: true
I20250115 14:52:10.864818 1167 ts_tablet_manager.cc:1905] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.864856 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f354e4204ec47149957eb22f964f08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.867401 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_af7ebd20849b40729c5650194d55240f" } modify_external_catalogs: true
I20250115 14:52:10.867527 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a11b7b0dd6494daf7b017edd4317a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.867602 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_af7ebd20849b40729c5650194d55240f" } modify_external_catalogs: true
I20250115 14:52:10.868485 882 tablet_service.cc:1467] Processing CreateTablet for tablet 51c3aa6f1d944ce782cee0507c7b1f6c (DEFAULT_TABLE table=default.Table_33d4b270a59a4b438798f77331accb24 [id=ebef90c0843f48c2aacd0c2446d250a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.868806 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c3aa6f1d944ce782cee0507c7b1f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.869035 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_35d4a8cd197f4504966cf56cc33f3a99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.869544 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35d4a8cd197f4504966cf56cc33f3a99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.869544 886 tablet_service.cc:1467] Processing CreateTablet for tablet 061234cb76b842a7a374ddd5f44fd9a1 (DEFAULT_TABLE table=default.Table_05f540a8c52d40b1b3240deb036757ad [id=c43c43f4d17f4e1ba7d6dacf6cd1dbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.869851 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061234cb76b842a7a374ddd5f44fd9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.870594 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c4f5e92e6f994cb0ad035e3cef60b3ce" } new_table_name: "default.Table_05b8a360f3f14e2ca92ea6d2a1567769" modify_external_catalogs: true
I20250115 14:52:10.870877 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 51c3aa6f1d944ce782cee0507c7b1f6c (DEFAULT_TABLE table=default.Table_33d4b270a59a4b438798f77331accb24 [id=ebef90c0843f48c2aacd0c2446d250a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.871165 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c3aa6f1d944ce782cee0507c7b1f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.877837 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0eeede8f714b92b1e3a9a558289c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.879913 893 tablet_service.cc:1467] Processing CreateTablet for tablet 1020437ffddf49cc846fcd3311892b46 (DEFAULT_TABLE table=default.Table_482c3b93feb244538f5483b3419a7a60 [id=a87ceba90988468da304990004f28635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.880247 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1020437ffddf49cc846fcd3311892b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.884790 1751 ts_tablet_manager.cc:1918] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:10.855769 737 tablet_service.cc:1467] Processing CreateTablet for tablet 1f354e4204ec47149957eb22f964f08a (DEFAULT_TABLE table=default.Table_f257047f59814e1a8458779252a782b9 [id=51d11f13250a43e8bc3b6988128d76d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.885309 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f354e4204ec47149957eb22f964f08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.885485 1751 log.cc:1198] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/1839edd0d49b467d8bb31b3001ff89eb
I20250115 14:52:10.889156 1751 ts_tablet_manager.cc:1939] T 1839edd0d49b467d8bb31b3001ff89eb P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:10.859855 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 84844b5f50af408ab7dda73f7421cd9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.860179 752 tablet_service.cc:1467] Processing CreateTablet for tablet 0936fa510a454b8e91d6ef2b345b18df (DEFAULT_TABLE table=default.Table_0610ab6883db4caa82a59fbf030bfedf [id=4c4b512016a24d24b6788075aeb0c3cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.877115 744 tablet_service.cc:1467] Processing CreateTablet for tablet 51c3aa6f1d944ce782cee0507c7b1f6c (DEFAULT_TABLE table=default.Table_33d4b270a59a4b438798f77331accb24 [id=ebef90c0843f48c2aacd0c2446d250a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.877621 741 tablet_service.cc:1467] Processing CreateTablet for tablet 061234cb76b842a7a374ddd5f44fd9a1 (DEFAULT_TABLE table=default.Table_05f540a8c52d40b1b3240deb036757ad [id=c43c43f4d17f4e1ba7d6dacf6cd1dbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.879594 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0eeede8f714b92b1e3a9a558289c3e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.891137 876 tablet_service.cc:1467] Processing CreateTablet for tablet 42d8c79e0f7a444c869378678344f2b7 (DEFAULT_TABLE table=default.Table_8bab10f63d3c43eca6cda9f4c2d4189a [id=9e41a8397db94127a19c471caaae75e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.892251 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42d8c79e0f7a444c869378678344f2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.884529 748 tablet_service.cc:1467] Processing CreateTablet for tablet 1020437ffddf49cc846fcd3311892b46 (DEFAULT_TABLE table=default.Table_482c3b93feb244538f5483b3419a7a60 [id=a87ceba90988468da304990004f28635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.890808 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 1020437ffddf49cc846fcd3311892b46 (DEFAULT_TABLE table=default.Table_482c3b93feb244538f5483b3419a7a60 [id=a87ceba90988468da304990004f28635]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.896139 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1020437ffddf49cc846fcd3311892b46. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:10.897333 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 356a0dfd7bb746548e0fb5b6a0935318 has no consensus state
I20250115 14:52:10.891934 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 0936fa510a454b8e91d6ef2b345b18df (DEFAULT_TABLE table=default.Table_0610ab6883db4caa82a59fbf030bfedf [id=4c4b512016a24d24b6788075aeb0c3cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.897763 447 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 5667ef3154c0487090e1d600d3763992
I20250115 14:52:10.898053 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0936fa510a454b8e91d6ef2b345b18df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.898119 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 8d4c30bccb3549e6963dfbbecf68cbbe
I20250115 14:52:10.899180 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4c30bccb3549e6963dfbbecf68cbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.898947 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 061234cb76b842a7a374ddd5f44fd9a1 (DEFAULT_TABLE table=default.Table_05f540a8c52d40b1b3240deb036757ad [id=c43c43f4d17f4e1ba7d6dacf6cd1dbd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.900574 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061234cb76b842a7a374ddd5f44fd9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.888710 1166 ts_tablet_manager.cc:1905] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.891737 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51c3aa6f1d944ce782cee0507c7b1f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.899121 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4c30bccb3549e6963dfbbecf68cbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.904439 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061234cb76b842a7a374ddd5f44fd9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.905484 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1020437ffddf49cc846fcd3311892b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.913355 1162 tablet_bootstrap.cc:492] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.913550 1162 ts_tablet_manager.cc:1397] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.087s user 0.002s sys 0.000s
I20250115 14:52:10.913633 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0936fa510a454b8e91d6ef2b345b18df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.914266 1162 raft_consensus.cc:357] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.914649 1162 raft_consensus.cc:383] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.914741 1162 raft_consensus.cc:738] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.915082 1162 consensus_queue.cc:260] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:10.918081 1162 ts_tablet_manager.cc:1428] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:10.918387 1162 tablet_bootstrap.cc:492] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:10.919705 1162 tablet_bootstrap.cc:654] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.924975 1168 tablet_bootstrap.cc:492] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:10.925137 1168 ts_tablet_manager.cc:1397] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.000s
I20250115 14:52:10.925810 1168 raft_consensus.cc:357] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.926018 1168 raft_consensus.cc:383] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.926079 1168 raft_consensus.cc:738] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.928656 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 1839edd0d49b467d8bb31b3001ff89eb (table default.Table_526b537d21ae4762b6881e46346414c6 [id=a724f6e4ad16485b8ee2445bb072be3e]) successfully deleted
I20250115 14:52:10.933984 1751 tablet_replica.cc:331] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:10.934244 1751 raft_consensus.cc:2238] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:10.934448 1751 raft_consensus.cc:2267] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:10.934657 2469 raft_consensus.cc:491] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.934772 2469 raft_consensus.cc:513] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.934970 1751 ts_tablet_manager.cc:1905] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:10.935287 2469 leader_election.cc:290] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.935379 2469 raft_consensus.cc:491] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.935451 2469 raft_consensus.cc:513] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.935953 2469 leader_election.cc:290] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:10.936542 1162 tablet_bootstrap.cc:492] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:10.936693 1162 ts_tablet_manager.cc:1397] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20250115 14:52:10.936944 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298fdd0217e4df6bfbafc49b2f2c65c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:10.937294 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248cadcdef8b4e99923cdbca39690685" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:10.937606 1162 raft_consensus.cc:357] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:10.937753 983 leader_election.cc:343] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.937810 1162 raft_consensus.cc:383] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:52:10.937945 983 leader_election.cc:343] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.937958 1162 raft_consensus.cc:738] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.938163 744 tablet_service.cc:1467] Processing CreateTablet for tablet 0b252e48108947cbbb36335499b51c92 (DEFAULT_TABLE table=default.Table_8ea6902fe95d4b84bdaf21fd1a87e38a [id=795c8172f7b946babddf8ee7c2707582]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:10.938282 1162 consensus_queue.cc:260] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.938555 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b252e48108947cbbb36335499b51c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:10.938910 1162 ts_tablet_manager.cc:1428] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:10.939253 1162 tablet_bootstrap.cc:492] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:10.942407 1162 tablet_bootstrap.cc:654] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.944721 1843 raft_consensus.cc:491] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.944845 1843 raft_consensus.cc:513] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.945345 1843 leader_election.cc:290] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:10.945639 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2484885df0da46f381d625a26e211c1e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:10.945894 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2484885df0da46f381d625a26e211c1e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:10.946046 1047 raft_consensus.cc:2463] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
W20250115 14:52:10.946159 848 leader_election.cc:343] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.946419 847 leader_election.cc:304] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.946810 1843 raft_consensus.cc:2798] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:10.946873 1843 raft_consensus.cc:491] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:10.946918 1843 raft_consensus.cc:3054] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:10.946904 1164 tablet_bootstrap.cc:492] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.947026 1164 ts_tablet_manager.cc:1397] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.146s user 0.003s sys 0.000s
I20250115 14:52:10.947715 1164 raft_consensus.cc:357] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.947870 1164 raft_consensus.cc:383] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.947917 1164 raft_consensus.cc:738] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.948223 1164 consensus_queue.cc:260] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.948681 1164 ts_tablet_manager.cc:1428] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:10.948938 1164 tablet_bootstrap.cc:492] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:10.949055 1843 raft_consensus.cc:513] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.949563 1843 leader_election.cc:290] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:10.949867 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2484885df0da46f381d625a26e211c1e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:10.950215 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2484885df0da46f381d625a26e211c1e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
W20250115 14:52:10.950330 848 leader_election.cc:343] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:10.950364 1047 raft_consensus.cc:3054] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:10.950600 1168 consensus_queue.cc:260] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.951078 1168 ts_tablet_manager.cc:1428] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250115 14:52:10.951314 1168 tablet_bootstrap.cc:492] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.952589 1168 tablet_bootstrap.cc:654] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.952759 1047 raft_consensus.cc:2463] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:10.953276 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d4c30bccb3549e6963dfbbecf68cbbe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.953423 847 leader_election.cc:304] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.953907 2340 raft_consensus.cc:2798] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:10.954012 2340 raft_consensus.cc:695] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:10.954216 2340 consensus_queue.cc:237] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.956403 454 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 92aeca346a7746468dbbff053d933472 (table default.Table_35a3371b3a6c4f35b2990f828671681e [id=60ff7f730d394990a0263e322d28d7ff]). Expected version 4 got 0
I20250115 14:52:10.958834 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 36de948083a946eba1a96f276e4f4f91 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:10.960058 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_19880e28f59446b0a4b4a8cc082428b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:10.960655 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19880e28f59446b0a4b4a8cc082428b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.964802 847 consensus_peers.cc:487] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 338c0f6f39fd4335bbcda733b28b8bf8. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:10.965487 1164 tablet_bootstrap.cc:654] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:10.965624 2341 raft_consensus.cc:491] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:10.965734 2341 raft_consensus.cc:513] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.966279 2341 leader_election.cc:290] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:10.966948 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377d711f052545ddbe499afe7ba3de28" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:10.967460 848 leader_election.cc:343] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:10.967767 847 leader_election.cc:343] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 377d711f052545ddbe499afe7ba3de28
I20250115 14:52:10.967409 1166 ts_tablet_manager.cc:1918] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.967904 847 leader_election.cc:304] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:10.968228 2341 raft_consensus.cc:2743] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:10.968569 1166 log.cc:1198] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/90c1602aef514dd29d5d2cffb043dbb6
I20250115 14:52:10.971503 1166 ts_tablet_manager.cc:1939] T 90c1602aef514dd29d5d2cffb043dbb6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:10.979069 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 638507d08f654011bea2597c22a788a2
I20250115 14:52:10.981271 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 42d8c79e0f7a444c869378678344f2b7
W20250115 14:52:10.982154 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d849c4e268a54191ae2632884f61edc0 consensus state has no leader
I20250115 14:52:10.982409 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d8c79e0f7a444c869378678344f2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.982419 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d8c79e0f7a444c869378678344f2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.982631 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 638507d08f654011bea2597c22a788a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:10.982879 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6cb8dfa7c42480bbd139d5acf059a57 consensus state has no leader
I20250115 14:52:10.983155 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 638507d08f654011bea2597c22a788a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:10.984026 1167 ts_tablet_manager.cc:1918] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.984128 1167 log.cc:1198] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/8442dc919d5f4d63b9b2ce7276cc0bc5
I20250115 14:52:10.984457 1167 ts_tablet_manager.cc:1939] T 8442dc919d5f4d63b9b2ce7276cc0bc5 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:10.984447 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet b76d1bbf90c6416ca60805c9d8053bd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e10fe1445be457a8c848c30771af866) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:10.985145 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 912) took 2512 ms. Trace:
I20250115 14:52:10.985229 1167 rpcz_store.cc:276] 0115 14:52:08.473033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.473491 (+ 458us) service_pool.cc:225] Handling call
0115 14:52:10.846697 (+2373206us) ts_tablet_manager.cc:1145] Deleting tablet 8442dc919d5f4d63b9b2ce7276cc0bc5
0115 14:52:10.846706 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:10.934884 (+ 88178us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.983921 (+ 49037us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:10.985126 (+ 1205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373117}
I20250115 14:52:10.985627 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 8442dc919d5f4d63b9b2ce7276cc0bc5 (table default.Table_b3a6b57353484cdf8f2dfde2738ae0c2 [id=5249bd4dd6f14f9bbfced76f1d336c71]) successfully deleted
I20250115 14:52:10.986809 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 42d8c79e0f7a444c869378678344f2b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.987455 1751 ts_tablet_manager.cc:1918] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:10.988271 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 90c1602aef514dd29d5d2cffb043dbb6 (table default.Table_950b48d1b1344479a7229041f044bbb6 [id=65eabb2be7d1479f889da99e50da551f]) successfully deleted
I20250115 14:52:10.988679 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 638507d08f654011bea2597c22a788a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:10.988966 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_01627515f7f64b1eb74dcf308bc61ea6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.989749 1168 tablet_bootstrap.cc:492] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:10.989686 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_030ac1279f7943c8bb81484f8da9a636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.989864 1168 ts_tablet_manager.cc:1397] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20250115 14:52:10.990331 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ea09a159f86444068a6a52f86e33ae66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.990509 1168 raft_consensus.cc:357] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.990665 1168 raft_consensus.cc:383] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.990713 1168 raft_consensus.cc:738] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.990934 1168 consensus_queue.cc:260] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.991221 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cc773ad5c57c4058baad0482e8dd2b79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:10.991410 1168 ts_tablet_manager.cc:1428] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250115 14:52:10.991655 1168 tablet_bootstrap.cc:492] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:10.992970 1168 tablet_bootstrap.cc:654] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:10.993026 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea09a159f86444068a6a52f86e33ae66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.994282 1164 tablet_bootstrap.cc:492] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:10.994627 1164 ts_tablet_manager.cc:1397] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
W20250115 14:52:10.995049 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_030ac1279f7943c8bb81484f8da9a636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:10.995391 1164 raft_consensus.cc:357] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:10.995630 1164 raft_consensus.cc:383] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:10.995685 1164 raft_consensus.cc:738] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:10.995898 1164 consensus_queue.cc:260] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:10.996142 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc773ad5c57c4058baad0482e8dd2b79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:10.988543 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3963e79868284401a915cac5ef258155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3963e79868284401a915cac5ef258155 already in progress: creating tablet
I20250115 14:52:11.001893 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_edbe5b64c3fb4732a52d6af29150486e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.002076 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01627515f7f64b1eb74dcf308bc61ea6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:11.002445 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edbe5b64c3fb4732a52d6af29150486e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.003240 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9cae3ab41ae3426081cc7cf3a3a05704"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.003698 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cae3ab41ae3426081cc7cf3a3a05704 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.004776 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_32080fc25c67476db569c3357f008e2c" } new_table_name: "default.Table_9105e59402544b38a833063271631821" modify_external_catalogs: true
I20250115 14:52:11.006042 1164 ts_tablet_manager.cc:1428] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250115 14:52:11.006345 1164 tablet_bootstrap.cc:492] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.007239 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b (table default.Table_ff30997993b446919e3eb0926e313f0f [id=174b82e1e06f4073a74a25aad8fc8045]). Expected version 1 got 0
I20250115 14:52:11.008019 1164 tablet_bootstrap.cc:654] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.010048 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 36de948083a946eba1a96f276e4f4f91
I20250115 14:52:11.010494 447 catalog_manager.cc:6596] ReplaceTablet: tablet 36de948083a946eba1a96f276e4f4f91 deleted and replaced by tablet c8e33ebec8b64f2398a7f1003b15cee7
I20250115 14:52:11.011330 2479 tablet.cc:1721] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250115 14:52:11.014293 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 36de948083a946eba1a96f276e4f4f91 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e33ebec8b64f2398a7f1003b15cee7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.015662 1167 tablet_replica.cc:331] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.015740 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b71be8b255d34a7b9b1e3fd33dc4fb8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.016242 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b71be8b255d34a7b9b1e3fd33dc4fb8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.020361 1167 raft_consensus.cc:2238] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.020521 1167 raft_consensus.cc:2267] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.021016 1167 ts_tablet_manager.cc:1905] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.022845 752 tablet_service.cc:1514] Processing DeleteTablet for tablet fa57c1fa96594a40b2551ad503127e4f with delete_type TABLET_DATA_DELETED (replaced by tablet 9c35882d2e5c464c9cfad496ba9d6af9) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.022933 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 36de948083a946eba1a96f276e4f4f91 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e33ebec8b64f2398a7f1003b15cee7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.024658 2480 tablet.cc:1721] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250115 14:52:11.033053 1751 log.cc:1198] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ca128bf3d6fc430298da932a78f54184
I20250115 14:52:11.035917 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 35301651222249d0b3439e0638f599f2 from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:11.035926 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 64a76d2772534687b1391e45b9cb984b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64a76d2772534687b1391e45b9cb984b already in progress: creating tablet
I20250115 14:52:10.994457 2273 raft_consensus.cc:491] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.036563 2273 raft_consensus.cc:513] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.036803 1751 ts_tablet_manager.cc:1939] T ca128bf3d6fc430298da932a78f54184 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:11.037173 2273 leader_election.cc:290] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.016650 2486 tablet.cc:1721] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250115 14:52:11.017333 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 36de948083a946eba1a96f276e4f4f91 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e33ebec8b64f2398a7f1003b15cee7) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:11.019939 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 85412.8us
I20250115 14:52:11.037707 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ca128bf3d6fc430298da932a78f54184 (table default.Table_8f53bbc08e5340aab38bb0ce8ef7aa16 [id=6cdcb9c1b34642d8a792807746ac26f5]) successfully deleted
I20250115 14:52:11.038015 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad988e25800f4af9a069b5a80582f749" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:11.038465 983 leader_election.cc:343] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.039006 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248cadcdef8b4e99923cdbca39690685" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:10.988862 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 0b252e48108947cbbb36335499b51c92 (DEFAULT_TABLE table=default.Table_8ea6902fe95d4b84bdaf21fd1a87e38a [id=795c8172f7b946babddf8ee7c2707582]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.039650 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b252e48108947cbbb36335499b51c92. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:11.042554 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2087b4ce466f4167b7c32c7074dcf6e7 consensus state has no leader
I20250115 14:52:11.044024 748 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a059e2f0654a4aafe9e8082b8fde7a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.044526 1166 tablet_replica.cc:331] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:11.048519 1166 raft_consensus.cc:2238] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.048785 1166 raft_consensus.cc:2267] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.048883 1168 tablet_bootstrap.cc:492] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.049024 1168 ts_tablet_manager.cc:1397] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.000s
I20250115 14:52:11.049649 1166 ts_tablet_manager.cc:1905] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.049675 1168 raft_consensus.cc:357] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.049844 1168 raft_consensus.cc:383] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.049898 1168 raft_consensus.cc:738] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.045379 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298fdd0217e4df6bfbafc49b2f2c65c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.050127 1168 consensus_queue.cc:260] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.046043 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad988e25800f4af9a069b5a80582f749" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.050624 1168 ts_tablet_manager.cc:1428] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:11.050879 1168 tablet_bootstrap.cc:492] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.052170 1168 tablet_bootstrap.cc:654] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.052367 2273 raft_consensus.cc:491] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.052506 2273 raft_consensus.cc:513] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.053107 2273 leader_election.cc:290] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.053208 2273 raft_consensus.cc:491] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.053292 2273 raft_consensus.cc:513] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.053766 2273 leader_election.cc:290] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:11.054260 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 13053264b5ca41dd85e53d928aa511fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13053264b5ca41dd85e53d928aa511fd already in progress: creating tablet
I20250115 14:52:11.054481 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.054838 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fe51b51994491ab38a644bfc55e3ac" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:11.055289 983 leader_election.cc:343] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:11.055466 983 leader_election.cc:343] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.056187 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fe51b51994491ab38a644bfc55e3ac" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.057211 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.058612 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_01c14574e8c94e2aac622d622f8ba52c" } new_table_name: "default.Table_493c7a30eba54405af8032e3f977a031" modify_external_catalogs: true
I20250115 14:52:11.059844 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_311231a4be29497898f249683c9fb37d" } modify_external_catalogs: true
I20250115 14:52:11.060029 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_311231a4be29497898f249683c9fb37d" } modify_external_catalogs: true
W20250115 14:52:11.062145 984 leader_election.cc:343] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.064106 875 tablet_service.cc:1467] Processing CreateTablet for tablet 0b252e48108947cbbb36335499b51c92 (DEFAULT_TABLE table=default.Table_8ea6902fe95d4b84bdaf21fd1a87e38a [id=795c8172f7b946babddf8ee7c2707582]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.064376 984 leader_election.cc:304] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.065457 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_92ea5828cb8a4d22b8965d819411004d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.065969 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92ea5828cb8a4d22b8965d819411004d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.063719 2516 raft_consensus.cc:491] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.064563 2486 tablet.cc:1721] T 05d590c63d6e4d058fac50dc50010317 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:11.066562 2516 raft_consensus.cc:513] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.067699 2516 leader_election.cc:290] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.069337 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_0d800cc4760a43fc82ef693a68ef4928 [id=be99ec07535243a68c1b3ee1be0c62e7] alter complete (version 4)
I20250115 14:52:11.071727 2273 raft_consensus.cc:2743] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.067560 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d850b85e948939008e5275d42e3cc" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:11.073189 983 leader_election.cc:343] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.073390 1167 ts_tablet_manager.cc:1918] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.073496 1167 log.cc:1198] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9e95d09efd2e44cc81cdcbf87ee453fe
W20250115 14:52:11.075065 984 leader_election.cc:343] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.075608 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b252e48108947cbbb36335499b51c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.077768 984 leader_election.cc:304] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.077839 1164 tablet_bootstrap.cc:492] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.077966 1164 ts_tablet_manager.cc:1397] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
W20250115 14:52:11.078171 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92eaeaf2dbd84a49b704e923743e07d0 consensus state has no leader
I20250115 14:52:11.078744 1164 raft_consensus.cc:357] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.078991 1164 raft_consensus.cc:383] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.079087 1164 raft_consensus.cc:738] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.079332 1164 consensus_queue.cc:260] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:11.079766 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6fd405050a2548e1b9ea7d7bf34a35af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fd405050a2548e1b9ea7d7bf34a35af already in progress: creating tablet
I20250115 14:52:11.079921 1164 ts_tablet_manager.cc:1428] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250115 14:52:11.080678 984 leader_election.cc:343] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.080845 984 leader_election.cc:304] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.081086 2469 raft_consensus.cc:2743] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.081395 2469 raft_consensus.cc:2743] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.081717 1166 ts_tablet_manager.cc:1918] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:11.081830 1166 log.cc:1198] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/c6f9262e808b4589a21257ed3fb0363b
I20250115 14:52:11.082166 1166 ts_tablet_manager.cc:1939] T c6f9262e808b4589a21257ed3fb0363b P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:11.082320 1164 tablet_bootstrap.cc:492] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.083714 1164 tablet_bootstrap.cc:654] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.084597 2324 raft_consensus.cc:491] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.084709 2324 raft_consensus.cc:513] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.085018 1162 tablet_bootstrap.cc:492] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:11.085140 1162 ts_tablet_manager.cc:1397] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.146s user 0.013s sys 0.029s
I20250115 14:52:11.085261 2324 leader_election.cc:290] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.085778 1162 raft_consensus.cc:357] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.085820 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84844b5f50af408ab7dda73f7421cd9d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.086014 1162 raft_consensus.cc:383] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.086005 909 raft_consensus.cc:2463] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:11.086129 1162 raft_consensus.cc:738] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.086393 1162 consensus_queue.cc:260] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.086462 710 leader_election.cc:304] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:11.086683 2324 raft_consensus.cc:2798] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:11.086752 2324 raft_consensus.cc:491] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:11.086789 1166 tablet_replica.cc:331] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:11.086820 2324 raft_consensus.cc:3054] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.086928 1162 ts_tablet_manager.cc:1428] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:11.087013 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]) successfully deleted
W20250115 14:52:11.087273 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9727430a15574ec0b24a5a64da744cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9727430a15574ec0b24a5a64da744cac already in progress: creating tablet
W20250115 14:52:11.087790 710 leader_election.cc:343] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 84844b5f50af408ab7dda73f7421cd9d
I20250115 14:52:11.088686 1166 raft_consensus.cc:2238] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.088862 1166 raft_consensus.cc:2267] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.089552 1166 ts_tablet_manager.cc:1905] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.089967 2324 raft_consensus.cc:513] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.090528 2324 leader_election.cc:290] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.091063 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84844b5f50af408ab7dda73f7421cd9d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:11.091229 909 raft_consensus.cc:3054] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.093869 909 raft_consensus.cc:2463] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:11.094349 710 leader_election.cc:304] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:11.094368 2324 raft_consensus.cc:491] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.094496 2324 raft_consensus.cc:513] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.094580 2403 raft_consensus.cc:2798] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:11.094663 2403 raft_consensus.cc:695] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:11.094857 2403 consensus_queue.cc:237] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.094971 2324 leader_election.cc:290] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.096376 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a000ab73dd46ac8e134784a156040a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:11.096560 710 leader_election.cc:343] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 07a000ab73dd46ac8e134784a156040a
I20250115 14:52:11.096596 909 raft_consensus.cc:2463] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:11.096930 710 leader_election.cc:304] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:11.097280 2495 raft_consensus.cc:2798] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:11.097339 2495 raft_consensus.cc:491] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:11.097389 2495 raft_consensus.cc:3054] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.099493 1166 ts_tablet_manager.cc:1918] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.099593 2495 raft_consensus.cc:513] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.100163 2495 leader_election.cc:290] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.100428 1751 tablet_replica.cc:331] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:11.100613 1751 raft_consensus.cc:2238] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.100776 1751 raft_consensus.cc:2267] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.101277 1751 ts_tablet_manager.cc:1905] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:11.101536 710 leader_election.cc:343] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 07a000ab73dd46ac8e134784a156040a
I20250115 14:52:11.101859 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07a000ab73dd46ac8e134784a156040a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:11.102007 909 raft_consensus.cc:3054] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.103374 2479 tablet.cc:1721] T 05d590c63d6e4d058fac50dc50010317 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:11.104481 909 raft_consensus.cc:2463] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:11.104810 710 leader_election.cc:304] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:11.104863 1162 tablet_bootstrap.cc:492] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:11.105151 2403 raft_consensus.cc:2798] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:11.105249 2403 raft_consensus.cc:695] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:11.105443 2403 consensus_queue.cc:237] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:11.105724 984 leader_election.cc:343] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.105830 984 leader_election.cc:304] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.106007 2516 raft_consensus.cc:2743] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.106200 1162 tablet_bootstrap.cc:654] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:11.106343 984 leader_election.cc:343] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.106427 984 leader_election.cc:304] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.106645 2516 raft_consensus.cc:2743] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.107656 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d850b85e948939008e5275d42e3cc" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.108865 2480 tablet.cc:1721] T 05d590c63d6e4d058fac50dc50010317 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:52:11.109637 710 leader_election.cc:343] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 84844b5f50af408ab7dda73f7421cd9d
I20250115 14:52:11.109710 2516 raft_consensus.cc:491] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.109834 2516 raft_consensus.cc:513] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.110364 2516 leader_election.cc:290] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.110919 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5265d5eeac83481ca97801e3475886da" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:11.111392 983 leader_election.cc:343] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.115245 1751 ts_tablet_manager.cc:1918] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:11.118986 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5265d5eeac83481ca97801e3475886da" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.120393 1751 log.cc:1198] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f3b75d759f834d558c94cc981e338ac0
I20250115 14:52:11.120847 1751 ts_tablet_manager.cc:1939] T f3b75d759f834d558c94cc981e338ac0 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:11.121969 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f3b75d759f834d558c94cc981e338ac0 (table default.Table_01bf4fd686f74e258977d847e2ee966e [id=f4f61992d591495f9e7a5a4b5c94fa32]) successfully deleted
I20250115 14:52:11.122360 1751 tablet_replica.cc:331] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:11.122592 1751 raft_consensus.cc:2238] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.122831 1751 raft_consensus.cc:2267] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.123384 1751 ts_tablet_manager.cc:1905] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.123785 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 4341977f363d40eeac71b7d33d1b43ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:11.122614 984 leader_election.cc:343] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.125111 984 leader_election.cc:304] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.126714 1162 tablet_bootstrap.cc:492] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:11.126893 1162 ts_tablet_manager.cc:1397] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.010s
I20250115 14:52:11.127790 2516 raft_consensus.cc:2743] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.128059 1162 raft_consensus.cc:357] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.128202 1162 raft_consensus.cc:383] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.128265 1162 raft_consensus.cc:738] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.128497 1162 consensus_queue.cc:260] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.129604 1162 ts_tablet_manager.cc:1428] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250115 14:52:11.129796 984 leader_election.cc:343] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.129904 1162 tablet_bootstrap.cc:492] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:11.129977 984 leader_election.cc:304] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.131268 1162 tablet_bootstrap.cc:654] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.131430 2469 raft_consensus.cc:2743] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.099583 1166 log.cc:1198] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/a600fa0f0ec049dca2803bf7d2c74a95
I20250115 14:52:11.133179 1166 ts_tablet_manager.cc:1939] T a600fa0f0ec049dca2803bf7d2c74a95 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:11.135190 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f79a705788cf4992a6b1be7d153f13f6" } new_table_name: "default.Table_7641cc01e3b0413a83202ed34b6a297e" modify_external_catalogs: true
I20250115 14:52:11.138761 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet a600fa0f0ec049dca2803bf7d2c74a95 (table default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d [id=ab53ab1ed1b34dafa524ccb35291fd62]) successfully deleted
I20250115 14:52:11.139446 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d70b1fff9275458798bb1b6bfd566a52" } modify_external_catalogs: true
I20250115 14:52:11.139619 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d70b1fff9275458798bb1b6bfd566a52" } modify_external_catalogs: true
I20250115 14:52:11.140100 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e7e9c0d93b9d4ec48187e83d5c70b6ac" } modify_external_catalogs: true
I20250115 14:52:11.140275 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e7e9c0d93b9d4ec48187e83d5c70b6ac" } modify_external_catalogs: true
I20250115 14:52:11.141640 1164 tablet_bootstrap.cc:492] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.141803 1164 ts_tablet_manager.cc:1397] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.000s
I20250115 14:52:11.141927 1167 ts_tablet_manager.cc:1939] T 9e95d09efd2e44cc81cdcbf87ee453fe P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
W20250115 14:52:11.142433 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29adc03e38624dfc910e07de24466d1b consensus state has no leader
I20250115 14:52:11.143113 2340 raft_consensus.cc:491] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.143234 2340 raft_consensus.cc:513] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.143828 2340 leader_election.cc:290] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:11.144961 847 leader_election.cc:343] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: ff9b57b8b9434f55ace5a4f47380df54
I20250115 14:52:11.146380 1168 tablet_bootstrap.cc:492] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.146502 1168 ts_tablet_manager.cc:1397] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.000s
I20250115 14:52:11.147104 1168 raft_consensus.cc:357] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.147274 1168 raft_consensus.cc:383] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.147327 1168 raft_consensus.cc:738] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.147504 1168 consensus_queue.cc:260] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.147923 1168 ts_tablet_manager.cc:1428] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:11.148166 1168 tablet_bootstrap.cc:492] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.149670 1168 tablet_bootstrap.cc:654] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.150727 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 913) took 2677 ms. Trace:
I20250115 14:52:11.150815 1167 rpcz_store.cc:276] 0115 14:52:08.473132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.473595 (+ 463us) service_pool.cc:225] Handling call
0115 14:52:11.015610 (+2542015us) ts_tablet_manager.cc:1145] Deleting tablet 9e95d09efd2e44cc81cdcbf87ee453fe
0115 14:52:11.015620 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.056138 (+ 40518us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.073297 (+ 17159us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.150708 (+ 77411us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2541927}
I20250115 14:52:11.151161 1167 tablet_replica.cc:331] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.151275 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9e95d09efd2e44cc81cdcbf87ee453fe (table default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]) successfully deleted
I20250115 14:52:11.151335 1167 raft_consensus.cc:2238] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.151464 1167 raft_consensus.cc:2267] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.151926 1167 ts_tablet_manager.cc:1905] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.143182 1166 tablet_replica.cc:331] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:11.150290 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9b57b8b9434f55ace5a4f47380df54" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.154147 1166 raft_consensus.cc:2238] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:11.154379 1166 pending_rounds.cc:62] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
W20250115 14:52:11.154428 848 leader_election.cc:343] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.154426 1166 pending_rounds.cc:69] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558386609774592 op_type: NO_OP noop_request { }
I20250115 14:52:11.154516 1166 raft_consensus.cc:2883] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:11.154516 848 leader_election.cc:304] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.154587 1166 raft_consensus.cc:2267] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.155433 1164 raft_consensus.cc:357] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.155591 1164 raft_consensus.cc:383] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.155635 1164 raft_consensus.cc:738] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.155668 1166 ts_tablet_manager.cc:1905] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.155841 1164 consensus_queue.cc:260] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.156397 1164 ts_tablet_manager.cc:1428] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:11.156558 2340 raft_consensus.cc:2743] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.161283 1751 ts_tablet_manager.cc:1918] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.161458 1751 log.cc:1198] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a48227ec5aa846c3ac63bd29ed38be25
I20250115 14:52:11.161787 1751 ts_tablet_manager.cc:1939] T a48227ec5aa846c3ac63bd29ed38be25 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:11.176039 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 0cae4c9228c640e495507e8f7ad9f544 from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:11.179343 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 92eaeaf2dbd84a49b704e923743e07d0 (table default.Table_493c7a30eba54405af8032e3f977a031 [id=2b2a775101124154bc8a37930c2a67d8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 92eaeaf2dbd84a49b704e923743e07d0 consensus state has no leader
I20250115 14:52:11.180763 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a48227ec5aa846c3ac63bd29ed38be25 (table default.Table_bb07dc16172246278fff8624127ab8bd [id=924019d3b4e842c2b3f173b307885151]) successfully deleted
I20250115 14:52:11.183719 1166 ts_tablet_manager.cc:1918] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:11.184036 1751 tablet_replica.cc:331] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:11.184278 1751 raft_consensus.cc:2238] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.184496 1751 raft_consensus.cc:2267] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.185071 1751 ts_tablet_manager.cc:1905] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.187291 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3592cd57b6b945288efcb39b7753ab52" } modify_external_catalogs: true
I20250115 14:52:11.187496 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3592cd57b6b945288efcb39b7753ab52" } modify_external_catalogs: true
I20250115 14:52:11.188109 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_169aa44347fe4b50b95e7c62b9604402" } modify_external_catalogs: true
I20250115 14:52:11.188273 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_169aa44347fe4b50b95e7c62b9604402" } modify_external_catalogs: true
I20250115 14:52:11.189338 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d02b19d9fa214584a8c9a02041a6da29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.189883 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d02b19d9fa214584a8c9a02041a6da29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.196084 1166 log.cc:1198] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/07a000ab73dd46ac8e134784a156040a
I20250115 14:52:11.188135 1164 tablet_bootstrap.cc:492] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.198024 1164 tablet_bootstrap.cc:654] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.198091 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1ba99d5a8ef746c3baa4c0cd075226e6" } new_table_name: "default.Table_0985c835daab4ee9944a45c2af4b3f91" modify_external_catalogs: true
I20250115 14:52:11.198714 1162 tablet_bootstrap.cc:492] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:11.198834 1162 ts_tablet_manager.cc:1397] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.013s
I20250115 14:52:11.199568 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2b06a00240dc4c3c9c3c257319305a60" } new_table_name: "default.Table_7accca692a1c423da80ab09d15b638ea" modify_external_catalogs: true
I20250115 14:52:11.199479 1162 raft_consensus.cc:357] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.207319 1162 raft_consensus.cc:383] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.207343 1168 tablet_bootstrap.cc:492] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.207398 1162 raft_consensus.cc:738] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.207475 1168 ts_tablet_manager.cc:1397] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.000s
I20250115 14:52:11.208096 1168 raft_consensus.cc:357] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.208261 1168 raft_consensus.cc:383] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.208319 1168 raft_consensus.cc:738] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.208493 1168 consensus_queue.cc:260] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.208968 1168 ts_tablet_manager.cc:1428] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250115 14:52:11.209208 1168 tablet_bootstrap.cc:492] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.207609 1162 consensus_queue.cc:260] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.211021 1162 ts_tablet_manager.cc:1428] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250115 14:52:11.212736 1166 ts_tablet_manager.cc:1939] T 07a000ab73dd46ac8e134784a156040a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:52:11.212917 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0414dcb8a44148a4ab5d7f5b15e9592d has no consensus state
W20250115 14:52:11.213392 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a005ad11e7941fab96b5565d55cfaf7 consensus state has no leader
W20250115 14:52:11.213817 1408 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:39235 (request call id 302) took 766 ms (client timeout 1000 ms). Trace:
I20250115 14:52:11.213973 1162 tablet_bootstrap.cc:492] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:11.213954 1408 rpcz_store.cc:269] 0115 14:52:10.442029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.454624 (+ 12595us) service_pool.cc:225] Handling call
0115 14:52:11.208750 (+754126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 14:52:11.216317 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c046302be54a4e91a532c737044d0f87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:11.216454 2469 raft_consensus.cc:491] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.216573 2469 raft_consensus.cc:513] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:11.216897 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c046302be54a4e91a532c737044d0f87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.217227 2469 leader_election.cc:290] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.219781 1166 tablet_replica.cc:331] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:11.219969 1166 raft_consensus.cc:2238] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:11.220207 1166 pending_rounds.cc:62] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
W20250115 14:52:11.221318 416 external_mini_cluster.cc:1677] Unable to check leaks on /tmp/dist-test-task__Z8ae/build/asan/bin/kudu pid 1221: Timed out: CheckLeaks RPC to 127.0.104.60:40227 timed out after 1.000s (SENT)
I20250115 14:52:11.221478 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task__Z8ae/build/asan/bin/kudu with pid 1221
I20250115 14:52:11.221907 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 07a000ab73dd46ac8e134784a156040a (table default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]) successfully deleted
I20250115 14:52:11.222422 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52fb251b0a48c2a8a4e07d55d22d3f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:11.222893 984 leader_election.cc:343] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.220248 1166 pending_rounds.cc:69] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558386566455296 op_type: NO_OP noop_request { }
I20250115 14:52:11.224391 1166 raft_consensus.cc:2883] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:11.224527 1166 raft_consensus.cc:2267] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.225664 1162 tablet_bootstrap.cc:654] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.226405 771 raft_consensus.cc:3054] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.218829 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52fb251b0a48c2a8a4e07d55d22d3f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.216677 1167 ts_tablet_manager.cc:1918] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.231586 1167 log.cc:1198] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/84844b5f50af408ab7dda73f7421cd9d
W20250115 14:52:11.231673 983 leader_election.cc:343] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.231781 983 leader_election.cc:304] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.231884 1167 ts_tablet_manager.cc:1939] T 84844b5f50af408ab7dda73f7421cd9d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.232167 2469 raft_consensus.cc:2743] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.232160 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 914) took 2668 ms. Trace:
I20250115 14:52:11.232237 1167 rpcz_store.cc:276] 0115 14:52:08.563233 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.563283 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:11.151119 (+2587836us) ts_tablet_manager.cc:1145] Deleting tablet 84844b5f50af408ab7dda73f7421cd9d
0115 14:52:11.151128 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.191716 (+ 40588us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.216574 (+ 24858us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.232145 (+ 15571us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2587671}
I20250115 14:52:11.232750 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 84844b5f50af408ab7dda73f7421cd9d (table default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]) successfully deleted
I20250115 14:52:11.233096 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 918) took 2586 ms. Trace:
I20250115 14:52:11.233170 1167 rpcz_store.cc:276] 0115 14:52:08.646479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.664383 (+ 17904us) service_pool.cc:225] Handling call
0115 14:52:11.232998 (+2568615us) ts_tablet_manager.cc:1145] Deleting tablet 32cb8768ca424aea87682295f30830a9
0115 14:52:11.233006 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.233082 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2568331}
W20250115 14:52:11.233526 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 32cb8768ca424aea87682295f30830a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32cb8768ca424aea87682295f30830a9 already in progress: creating tablet
I20250115 14:52:11.233841 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 930) took 2416 ms. Trace:
I20250115 14:52:11.233912 1167 rpcz_store.cc:276] 0115 14:52:08.817180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.823173 (+ 5993us) service_pool.cc:225] Handling call
0115 14:52:11.233750 (+2410577us) ts_tablet_manager.cc:1145] Deleting tablet 6203efb862ea4962a6d09f3e4306a92f
0115 14:52:11.233759 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.233827 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2410369}
W20250115 14:52:11.234261 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6203efb862ea4962a6d09f3e4306a92f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6203efb862ea4962a6d09f3e4306a92f already in progress: creating tablet
I20250115 14:52:11.234549 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 933) took 2293 ms. Trace:
I20250115 14:52:11.234619 1167 rpcz_store.cc:276] 0115 14:52:08.941300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.941355 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:11.234466 (+2293111us) ts_tablet_manager.cc:1145] Deleting tablet d18a88c8f3ce4a24b7e1e13966760364
0115 14:52:11.234473 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.234536 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2292921}
W20250115 14:52:11.234956 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d18a88c8f3ce4a24b7e1e13966760364 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d18a88c8f3ce4a24b7e1e13966760364 already in progress: creating tablet
I20250115 14:52:11.235235 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 932) took 2296 ms. Trace:
I20250115 14:52:11.235299 1167 rpcz_store.cc:276] 0115 14:52:08.939215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:08.954669 (+ 15454us) service_pool.cc:225] Handling call
0115 14:52:11.235152 (+2280483us) ts_tablet_manager.cc:1145] Deleting tablet ff771f5acdcf4fcb9e099aede920f0a1
0115 14:52:11.235160 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.235222 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2280019}
W20250115 14:52:11.235621 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ff771f5acdcf4fcb9e099aede920f0a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff771f5acdcf4fcb9e099aede920f0a1 already in progress: creating tablet
I20250115 14:52:11.235901 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 936) took 2148 ms. Trace:
I20250115 14:52:11.235965 1167 rpcz_store.cc:276] 0115 14:52:09.087091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.087138 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:11.235816 (+2148678us) ts_tablet_manager.cc:1145] Deleting tablet f2ba4086a6bf4bfa83e740b31dfb53f8
0115 14:52:11.235824 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.235887 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2148516}
I20250115 14:52:11.236714 1166 ts_tablet_manager.cc:1905] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:11.237711 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f2ba4086a6bf4bfa83e740b31dfb53f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2ba4086a6bf4bfa83e740b31dfb53f8 already in progress: creating tablet
I20250115 14:52:11.238374 1167 tablet_replica.cc:331] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.238406 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f6cc9ac2a299480b990c0a3ddcf355fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:11.238554 1167 raft_consensus.cc:2238] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:11.238830 1167 raft_consensus.cc:2267] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:11.238900 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6cc9ac2a299480b990c0a3ddcf355fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.239288 1167 ts_tablet_manager.cc:1905] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.210721 1168 tablet_bootstrap.cc:654] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.242903 1164 tablet_bootstrap.cc:492] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.243013 1164 ts_tablet_manager.cc:1397] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20250115 14:52:11.243579 1164 raft_consensus.cc:357] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.243708 2340 raft_consensus.cc:491] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.243744 1164 raft_consensus.cc:383] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.243800 1164 raft_consensus.cc:738] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.243813 2340 raft_consensus.cc:513] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.243981 1164 consensus_queue.cc:260] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.244431 2340 leader_election.cc:290] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.244597 1164 ts_tablet_manager.cc:1428] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:11.244913 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6942d102069d4c33b23fa2c0b29f94a6" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.245218 1164 tablet_bootstrap.cc:492] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:11.245384 848 leader_election.cc:343] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:11.246114 847 leader_election.cc:343] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6942d102069d4c33b23fa2c0b29f94a6
I20250115 14:52:11.246214 847 leader_election.cc:304] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.246619 2340 raft_consensus.cc:2743] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.246793 2480 tablet.cc:1721] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:11.253727 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8e7481bab16b4d808c56d4a6b10c11e5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.254285 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e7481bab16b4d808c56d4a6b10c11e5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.256531 2469 raft_consensus.cc:491] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.256669 2469 raft_consensus.cc:513] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.256832 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_32d4e94660334e308afef5563b749191"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:11.257280 2469 leader_election.cc:290] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:11.257308 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32d4e94660334e308afef5563b749191 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.257952 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70425c25919b4d0b999450210e1cb33f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.258389 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70425c25919b4d0b999450210e1cb33f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:11.258518 983 leader_election.cc:343] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:11.258817 984 leader_election.cc:343] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.258906 984 leader_election.cc:304] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.259265 2469 raft_consensus.cc:2743] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.259433 1162 tablet_bootstrap.cc:492] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:11.259588 1162 ts_tablet_manager.cc:1397] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.002s
I20250115 14:52:11.260277 1162 raft_consensus.cc:357] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.264675 1167 ts_tablet_manager.cc:1918] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:11.264794 1167 log.cc:1198] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/be2fd31f74a94787a5e8ccae3c5ef245
I20250115 14:52:11.265139 1168 tablet_bootstrap.cc:492] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.265305 1168 ts_tablet_manager.cc:1397] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20250115 14:52:11.265316 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_08ec29c2d0ca4d7db6cd48a97784dced"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.265822 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08ec29c2d0ca4d7db6cd48a97784dced in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.266036 1168 raft_consensus.cc:357] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.266203 1168 raft_consensus.cc:383] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.266243 1168 raft_consensus.cc:738] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.266417 1168 consensus_queue.cc:260] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.266885 1168 ts_tablet_manager.cc:1428] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:11.267376 1167 ts_tablet_manager.cc:1939] T be2fd31f74a94787a5e8ccae3c5ef245 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.267413 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5fc99dfa3f504b2f9a09dcfc542dde73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:11.268543 1168 tablet_bootstrap.cc:492] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:11.269320 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fc99dfa3f504b2f9a09dcfc542dde73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.269631 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 937) took 2106 ms. Trace:
I20250115 14:52:11.269712 1167 rpcz_store.cc:276] 0115 14:52:09.162711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.162797 (+ 86us) service_pool.cc:225] Handling call
0115 14:52:11.238335 (+2075538us) ts_tablet_manager.cc:1145] Deleting tablet be2fd31f74a94787a5e8ccae3c5ef245
0115 14:52:11.238342 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.251344 (+ 13002us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.264585 (+ 13241us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.269617 (+ 5032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2045914}
I20250115 14:52:11.270124 1168 tablet_bootstrap.cc:654] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.270152 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet be2fd31f74a94787a5e8ccae3c5ef245 (table default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec [id=724c015c8f3f4aa581ac6faa84187b06]) successfully deleted
I20250115 14:52:11.270923 1162 raft_consensus.cc:383] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.271073 1162 raft_consensus.cc:738] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.271378 1162 consensus_queue.cc:260] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.272684 1162 ts_tablet_manager.cc:1428] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20250115 14:52:11.273659 1166 ts_tablet_manager.cc:1918] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:11.274010 1162 tablet_bootstrap.cc:492] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:11.281684 1162 tablet_bootstrap.cc:654] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.281872 1166 log.cc:1198] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/84844b5f50af408ab7dda73f7421cd9d
I20250115 14:52:11.282298 1166 ts_tablet_manager.cc:1939] T 84844b5f50af408ab7dda73f7421cd9d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:11.282590 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 92aeca346a7746468dbbff053d933472 (table default.Table_35a3371b3a6c4f35b2990f828671681e [id=60ff7f730d394990a0263e322d28d7ff]). Expected version 4 got 0
I20250115 14:52:11.282857 447 catalog_manager.cc:5526] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:11.283357 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 2484885df0da46f381d625a26e211c1e (table default.Table_8652d6d9ec084553b5183aa372c2554a [id=c29c4105b83d4df9b617997c0c9c5b14]). Expected version 1 got 0
I20250115 14:52:11.284617 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 84844b5f50af408ab7dda73f7421cd9d (table default.Table_2989f3ca246d42528088a1e17c79eb91 [id=859ca28f28914045b1ca3ed39e0680be]) successfully deleted
I20250115 14:52:11.285369 1167 tablet_replica.cc:331] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.285565 1167 raft_consensus.cc:2238] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.285742 1167 raft_consensus.cc:2267] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:11.286345 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5f0eeede8f714b92b1e3a9a558289c3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f0eeede8f714b92b1e3a9a558289c3e already in progress: creating tablet
W20250115 14:52:11.286903 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 8d4c30bccb3549e6963dfbbecf68cbbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d4c30bccb3549e6963dfbbecf68cbbe already in progress: creating tablet
W20250115 14:52:11.287600 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 42d8c79e0f7a444c869378678344f2b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42d8c79e0f7a444c869378678344f2b7 already in progress: creating tablet
I20250115 14:52:11.287914 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 288b0e455a854464940f65a2ab909d0b
I20250115 14:52:11.288617 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 288b0e455a854464940f65a2ab909d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.288630 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ff30997993b446919e3eb0926e313f0f" } new_table_name: "default.Table_f634603ea42b4215b8ef3df8c446ada7" modify_external_catalogs: true
I20250115 14:52:11.289115 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 288b0e455a854464940f65a2ab909d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.289863 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 288b0e455a854464940f65a2ab909d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.289942 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_c1296b8b98f242948cfbcea7f0158ee5 [id=ef96f8cc27264b11803d888505942ae7] alter complete (version 1)
I20250115 14:52:11.290382 1167 ts_tablet_manager.cc:1905] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.290827 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_42e1685088164128aefa917c691bbf7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.291338 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42e1685088164128aefa917c691bbf7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.291983 748 tablet_service.cc:1467] Processing CreateTablet for tablet 700ec53060f3422aa2ea02ea6426f3f6 (DEFAULT_TABLE table=default.Table_220ee236fc72425ab7480c72858f2989 [id=b1a5deeacdb44a089831fca5ce3ee6ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.292382 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 700ec53060f3422aa2ea02ea6426f3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.292426 1166 tablet_replica.cc:331] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:11.292496 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 325cd3fd8b4d447998822163e575e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.292577 1166 raft_consensus.cc:2238] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.292704 1166 raft_consensus.cc:2267] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.293439 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c172b6003608480ca15c2e050f16e291"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.293900 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c172b6003608480ca15c2e050f16e291 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.294329 1166 ts_tablet_manager.cc:1905] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.295425 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 3963e79868284401a915cac5ef258155 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:11.295979 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_15364f05c206457da9c472858167e5dd" } new_table_name: "default.Table_f9607243e98b4e9599bb43a57373370f" modify_external_catalogs: true
I20250115 14:52:11.299947 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a000ab73dd46ac8e134784a156040a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:57 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:11.300158 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 638507d08f654011bea2597c22a788a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 638507d08f654011bea2597c22a788a2 already in progress: creating tablet
I20250115 14:52:11.300242 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 84844b5f50af408ab7dda73f7421cd9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.262848 1751 ts_tablet_manager.cc:1918] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.301651 1751 log.cc:1198] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6e616d78933e4f84b9ec2acc51450b18
I20250115 14:52:11.302008 1751 ts_tablet_manager.cc:1939] T 6e616d78933e4f84b9ec2acc51450b18 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:11.302261 875 tablet_service.cc:1467] Processing CreateTablet for tablet 700ec53060f3422aa2ea02ea6426f3f6 (DEFAULT_TABLE table=default.Table_220ee236fc72425ab7480c72858f2989 [id=b1a5deeacdb44a089831fca5ce3ee6ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.303010 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6e616d78933e4f84b9ec2acc51450b18 (table default.Table_ff33645127a443efb2b776d808942dba [id=f8c4ca50220a4dbf9b9480213d836e5f]) successfully deleted
I20250115 14:52:11.304090 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 700ec53060f3422aa2ea02ea6426f3f6 (DEFAULT_TABLE table=default.Table_220ee236fc72425ab7480c72858f2989 [id=b1a5deeacdb44a089831fca5ce3ee6ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.304454 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 700ec53060f3422aa2ea02ea6426f3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.305311 876 tablet_service.cc:1467] Processing CreateTablet for tablet c9a871e5d92d40c09042323a00453a0e (DEFAULT_TABLE table=default.Table_23055a20564d4db3b53d4876e1995d07 [id=a14308ef0dbf4f1fb5ba4a58479027b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.306857 744 tablet_service.cc:1467] Processing CreateTablet for tablet c9a871e5d92d40c09042323a00453a0e (DEFAULT_TABLE table=default.Table_23055a20564d4db3b53d4876e1995d07 [id=a14308ef0dbf4f1fb5ba4a58479027b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.306962 752 tablet_service.cc:1467] Processing CreateTablet for tablet 9022738b53c74ce99e8f35adb655d200 (DEFAULT_TABLE table=default.Table_6c9c88e11c6d4580889dd2b45811a57f [id=f0a8375a7f7049c595328c6a9bd65c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.307209 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a871e5d92d40c09042323a00453a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.307425 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9022738b53c74ce99e8f35adb655d200. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.309804 893 tablet_service.cc:1467] Processing CreateTablet for tablet 9022738b53c74ce99e8f35adb655d200 (DEFAULT_TABLE table=default.Table_6c9c88e11c6d4580889dd2b45811a57f [id=f0a8375a7f7049c595328c6a9bd65c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.310608 1015 tablet_service.cc:1467] Processing CreateTablet for tablet c9a871e5d92d40c09042323a00453a0e (DEFAULT_TABLE table=default.Table_23055a20564d4db3b53d4876e1995d07 [id=a14308ef0dbf4f1fb5ba4a58479027b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.310978 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a871e5d92d40c09042323a00453a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.314278 879 tablet_service.cc:1467] Processing CreateTablet for tablet a1efe163729f4097979333257f2eabcc (DEFAULT_TABLE table=default.Table_8e2ac157ed9045618562bb7d16dd9aaa [id=4005bc40261e404e84bf3a4c6e4d7263]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.315531 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 9022738b53c74ce99e8f35adb655d200 (DEFAULT_TABLE table=default.Table_6c9c88e11c6d4580889dd2b45811a57f [id=f0a8375a7f7049c595328c6a9bd65c18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.315893 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9022738b53c74ce99e8f35adb655d200. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.317416 1014 tablet_service.cc:1467] Processing CreateTablet for tablet a1efe163729f4097979333257f2eabcc (DEFAULT_TABLE table=default.Table_8e2ac157ed9045618562bb7d16dd9aaa [id=4005bc40261e404e84bf3a4c6e4d7263]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.317724 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1efe163729f4097979333257f2eabcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.318665 752 tablet_service.cc:1467] Processing CreateTablet for tablet a1efe163729f4097979333257f2eabcc (DEFAULT_TABLE table=default.Table_8e2ac157ed9045618562bb7d16dd9aaa [id=4005bc40261e404e84bf3a4c6e4d7263]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.319006 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1efe163729f4097979333257f2eabcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.321267 880 tablet_service.cc:1467] Processing CreateTablet for tablet 13a4837b72eb4008ad612a9fc86dfac7 (DEFAULT_TABLE table=default.Table_5b2238ba86fe41079543db9d13c4c1f4 [id=87305597cbdb42d595750887c01ea856]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.322381 744 tablet_service.cc:1467] Processing CreateTablet for tablet 13a4837b72eb4008ad612a9fc86dfac7 (DEFAULT_TABLE table=default.Table_5b2238ba86fe41079543db9d13c4c1f4 [id=87305597cbdb42d595750887c01ea856]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.322911 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13a4837b72eb4008ad612a9fc86dfac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.324551 1751 tablet_replica.cc:331] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:11.324808 1751 raft_consensus.cc:2238] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.324954 1751 raft_consensus.cc:2267] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.325451 1751 ts_tablet_manager.cc:1905] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.331398 449 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 246720dbb4ed4216ac87fb4a3df79e2c (table default.Table_daaa2755fe9d4f54ae6ff33ae452ec05 [id=8c0229e6f6884ea7be3417678bb27377]). Expected version 1 got 0
I20250115 14:52:11.333628 886 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe7404aa6ed49ca83d53d5d6e7d532c (DEFAULT_TABLE table=default.Table_f3626b709cb3478597936e4d350b38a1 [id=60785a1ed5fb4f54b29d5947a6f9f179]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.333813 1164 tablet_bootstrap.cc:654] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.336105 2340 raft_consensus.cc:491] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.336232 2340 raft_consensus.cc:513] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.336902 2340 leader_election.cc:290] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.337391 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742792be08444792892bd9cd7336dda3" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:11.337941 848 leader_election.cc:343] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.339035 752 tablet_service.cc:1467] Processing CreateTablet for tablet 234fd4e63b554cb583ef3df9dd64385c (DEFAULT_TABLE table=default.Table_f436acfa7f9d4abd8d6ca5f71c593faa [id=7fd95155ec424914992f1d2bf7f1453f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.339363 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234fd4e63b554cb583ef3df9dd64385c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.340368 882 tablet_service.cc:1467] Processing CreateTablet for tablet 234fd4e63b554cb583ef3df9dd64385c (DEFAULT_TABLE table=default.Table_f436acfa7f9d4abd8d6ca5f71c593faa [id=7fd95155ec424914992f1d2bf7f1453f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.340864 748 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe7404aa6ed49ca83d53d5d6e7d532c (DEFAULT_TABLE table=default.Table_f3626b709cb3478597936e4d350b38a1 [id=60785a1ed5fb4f54b29d5947a6f9f179]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.342904 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe7404aa6ed49ca83d53d5d6e7d532c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.343684 741 tablet_service.cc:1514] Processing DeleteTablet for tablet be2fd31f74a94787a5e8ccae3c5ef245 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.343710 740 tablet_service.cc:1467] Processing CreateTablet for tablet a696017e859f474bb1fd062df44c224d (DEFAULT_TABLE table=default.Table_e7e9c0d93b9d4ec48187e83d5c70b6ac [id=eeea3b3c85684c1a86aeb2050609e4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.343981 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a696017e859f474bb1fd062df44c224d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.343994 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234fd4e63b554cb583ef3df9dd64385c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.344203 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe7404aa6ed49ca83d53d5d6e7d532c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.344465 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 700ec53060f3422aa2ea02ea6426f3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.345010 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13a4837b72eb4008ad612a9fc86dfac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.348551 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9022738b53c74ce99e8f35adb655d200. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.348711 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9a871e5d92d40c09042323a00453a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.348927 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1efe163729f4097979333257f2eabcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.358570 1166 ts_tablet_manager.cc:1918] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.358695 1166 log.cc:1198] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/fa57c1fa96594a40b2551ad503127e4f
I20250115 14:52:11.358965 1166 ts_tablet_manager.cc:1939] T fa57c1fa96594a40b2551ad503127e4f P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:11.360212 741 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a11b7b0dd6494daf7b017edd4317a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.360460 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7a7640e000334453a1519964f3fee50c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:11.361074 1164 tablet_bootstrap.cc:492] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
W20250115 14:52:11.361183 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a7640e000334453a1519964f3fee50c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.361241 1164 ts_tablet_manager.cc:1397] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.001s
I20250115 14:52:11.362277 1164 raft_consensus.cc:357] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.362516 1164 raft_consensus.cc:383] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.362597 1164 raft_consensus.cc:738] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.362839 1164 consensus_queue.cc:260] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.365460 893 tablet_service.cc:1467] Processing CreateTablet for tablet a696017e859f474bb1fd062df44c224d (DEFAULT_TABLE table=default.Table_e7e9c0d93b9d4ec48187e83d5c70b6ac [id=eeea3b3c85684c1a86aeb2050609e4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.365561 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a696017e859f474bb1fd062df44c224d (DEFAULT_TABLE table=default.Table_e7e9c0d93b9d4ec48187e83d5c70b6ac [id=eeea3b3c85684c1a86aeb2050609e4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.365816 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a696017e859f474bb1fd062df44c224d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.365936 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a696017e859f474bb1fd062df44c224d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.366941 1164 ts_tablet_manager.cc:1428] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250115 14:52:11.373726 751 tablet_service.cc:1467] Processing CreateTablet for tablet 732708e96ddc477bbf9314cce3b78ddb (DEFAULT_TABLE table=default.Table_ac9b6dbf8a094c10aa1277694fc74c2a [id=c55656dbe6b342b7be8c366ad47b4f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.374112 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 732708e96ddc477bbf9314cce3b78ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.377413 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 356a0dfd7bb746548e0fb5b6a0935318 (DEFAULT_TABLE table=default.Table_c49292ffc41542abb7c1104187ce5cba [id=085b32beb8424ec98e18d011609b6629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.377753 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 356a0dfd7bb746548e0fb5b6a0935318. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.377895 741 tablet_service.cc:1467] Processing CreateTablet for tablet 356a0dfd7bb746548e0fb5b6a0935318 (DEFAULT_TABLE table=default.Table_c49292ffc41542abb7c1104187ce5cba [id=085b32beb8424ec98e18d011609b6629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.378214 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 356a0dfd7bb746548e0fb5b6a0935318. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.378476 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 732708e96ddc477bbf9314cce3b78ddb (DEFAULT_TABLE table=default.Table_ac9b6dbf8a094c10aa1277694fc74c2a [id=c55656dbe6b342b7be8c366ad47b4f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.378754 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 732708e96ddc477bbf9314cce3b78ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.379202 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb3952b65cd477783c0425984bac152 (DEFAULT_TABLE table=default.Table_b72e84abd6a046eeb0e832a67bc06490 [id=ac4f137594bd4b018e0816c3353fe50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.382525 893 tablet_service.cc:1467] Processing CreateTablet for tablet 44c70b9eba794f6a8ee393679db01a4c (DEFAULT_TABLE table=default.Table_4a7ea7c535504cfdb8ba930aa3f60055 [id=2a5d91dab0674283bd4338de14c96642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.382877 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c70b9eba794f6a8ee393679db01a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.383160 891 tablet_service.cc:1467] Processing CreateTablet for tablet 96692985a2c240099dbcb191fbd22384 (DEFAULT_TABLE table=default.Table_d100b9e08d3045d083f5800a8d651450 [id=c64973aed838415d9a339344fe777e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.383282 886 tablet_service.cc:1467] Processing CreateTablet for tablet 4968ce56c6b4457097880a8ea7f924d8 (DEFAULT_TABLE table=default.Table_da2d5c7907c24047bd4f7c0f6b70aa45 [id=5ba88bc9fd1b40a881f2560c8271a49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.383472 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96692985a2c240099dbcb191fbd22384. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.383733 880 tablet_service.cc:1467] Processing CreateTablet for tablet ace1012dde2b4ecbb9e58e1ad608644f (DEFAULT_TABLE table=default.Table_0ac18ecc16b2424881834e142ba358d9 [id=c76d044827ce4916b5f914622df4f9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.384023 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ace1012dde2b4ecbb9e58e1ad608644f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.385406 740 tablet_service.cc:1467] Processing CreateTablet for tablet cbcd9fc1ed6d46209d8482c39ec6c3ba (DEFAULT_TABLE table=default.Table_05b6514d0e1a4ee2941c4f2b693bf29d [id=beb779f0483a410596b30bb5f1f0c8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.385705 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbcd9fc1ed6d46209d8482c39ec6c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.385732 756 tablet_service.cc:1467] Processing CreateTablet for tablet 96692985a2c240099dbcb191fbd22384 (DEFAULT_TABLE table=default.Table_d100b9e08d3045d083f5800a8d651450 [id=c64973aed838415d9a339344fe777e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.386023 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96692985a2c240099dbcb191fbd22384. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.387454 755 tablet_service.cc:1467] Processing CreateTablet for tablet 4968ce56c6b4457097880a8ea7f924d8 (DEFAULT_TABLE table=default.Table_da2d5c7907c24047bd4f7c0f6b70aa45 [id=5ba88bc9fd1b40a881f2560c8271a49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.387727 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4968ce56c6b4457097880a8ea7f924d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.388510 885 tablet_service.cc:1467] Processing CreateTablet for tablet 928e7cd1e2694d548bc973800f99796c (DEFAULT_TABLE table=default.Table_c7660a4693d04427a14694d941c8f631 [id=e28b2d17b9534b3cbf309ab00898ab9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.388828 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 928e7cd1e2694d548bc973800f99796c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.389039 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2ebae7ff1b0b4ce6b558bf9a28b8fa72 (DEFAULT_TABLE table=default.Table_1d252b88a14246be9189c5c5261887c9 [id=7519c84d120b46959c1d6ec3b79d1a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.389315 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebae7ff1b0b4ce6b558bf9a28b8fa72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.390974 879 tablet_service.cc:1467] Processing CreateTablet for tablet 356a0dfd7bb746548e0fb5b6a0935318 (DEFAULT_TABLE table=default.Table_c49292ffc41542abb7c1104187ce5cba [id=085b32beb8424ec98e18d011609b6629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.391254 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 356a0dfd7bb746548e0fb5b6a0935318. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.392459 882 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb3952b65cd477783c0425984bac152 (DEFAULT_TABLE table=default.Table_b72e84abd6a046eeb0e832a67bc06490 [id=ac4f137594bd4b018e0816c3353fe50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.392505 737 tablet_service.cc:1467] Processing CreateTablet for tablet ace1012dde2b4ecbb9e58e1ad608644f (DEFAULT_TABLE table=default.Table_0ac18ecc16b2424881834e142ba358d9 [id=c76d044827ce4916b5f914622df4f9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.392781 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb3952b65cd477783c0425984bac152. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.392796 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ace1012dde2b4ecbb9e58e1ad608644f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.396499 753 tablet_service.cc:1467] Processing CreateTablet for tablet 9cb3952b65cd477783c0425984bac152 (DEFAULT_TABLE table=default.Table_b72e84abd6a046eeb0e832a67bc06490 [id=ac4f137594bd4b018e0816c3353fe50f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.396791 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb3952b65cd477783c0425984bac152. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.401599 740 tablet_service.cc:1467] Processing CreateTablet for tablet 7539a25fbf6c4e28b5538c8a29eaf529 (DEFAULT_TABLE table=default.Table_01bf4fd686f74e258977d847e2ee966e [id=f4f61992d591495f9e7a5a4b5c94fa32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.401707 749 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2b1460d93e4c99afb45daf5e7d138e (DEFAULT_TABLE table=default.Table_e55f8e94ee304fd9ab86502d92eecf50 [id=239cde4385364164917f1181ee951874]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.401891 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7539a25fbf6c4e28b5538c8a29eaf529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.402074 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a2b1460d93e4c99afb45daf5e7d138e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.402216 1166 tablet_replica.cc:331] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:11.402383 1166 raft_consensus.cc:2238] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.402513 1166 raft_consensus.cc:2267] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.404263 745 tablet_service.cc:1467] Processing CreateTablet for tablet 928e7cd1e2694d548bc973800f99796c (DEFAULT_TABLE table=default.Table_c7660a4693d04427a14694d941c8f631 [id=e28b2d17b9534b3cbf309ab00898ab9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.404552 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 928e7cd1e2694d548bc973800f99796c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.407186 1166 ts_tablet_manager.cc:1905] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.408519 747 tablet_service.cc:1467] Processing CreateTablet for tablet b7c6c8d5c58142c0af3e726ea57207b9 (DEFAULT_TABLE table=default.Table_10d8c601efc24a47b14b76858704b724 [id=d12776269f4045f78a988b1d8a56e691]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.408483 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet fa57c1fa96594a40b2551ad503127e4f (table default.Table_958850c3f0a54c49a8dd42b180068897 [id=8242d7a08fcd4d4e836f6b70b1bacdc1]) successfully deleted
I20250115 14:52:11.408846 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c6c8d5c58142c0af3e726ea57207b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:11.409087 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 36de948083a946eba1a96f276e4f4f91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36de948083a946eba1a96f276e4f4f91 already in progress: creating tablet
I20250115 14:52:11.410570 750 tablet_service.cc:1467] Processing CreateTablet for tablet 17aa6950326d4226976e07c7cc986d39 (DEFAULT_TABLE table=default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.410851 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17aa6950326d4226976e07c7cc986d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.411633 756 tablet_service.cc:1467] Processing CreateTablet for tablet 45204abca48c42ff82f6882532142be4 (DEFAULT_TABLE table=default.Table_a18f52945a28410fb91e6dfeef1cdcb9 [id=028275009f044112bd6b65996d609b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.412137 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45204abca48c42ff82f6882532142be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.413110 743 tablet_service.cc:1467] Processing CreateTablet for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b (DEFAULT_TABLE table=default.Table_311231a4be29497898f249683c9fb37d [id=fcb55e82b323456aa31cb8d89a1b01b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.413388 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.415827 750 tablet_service.cc:1467] Processing CreateTablet for tablet 237f3ec8c48a48bc975922f234fb3950 (DEFAULT_TABLE table=default.Table_c1102208539341c0a46b3222273132f3 [id=7934961e2c63425ca064f91c7073ed38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.416105 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 237f3ec8c48a48bc975922f234fb3950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.416529 742 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed31458aff04da0b1287c932a060ded (DEFAULT_TABLE table=default.Table_a0baa54d6f6f437d954be2ba0f6c03d9 [id=4e9ff162dda6488baf39463c68228a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.416807 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed31458aff04da0b1287c932a060ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.416759 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0414dcb8a44148a4ab5d7f5b15e9592d (DEFAULT_TABLE table=default.Table_7accca692a1c423da80ab09d15b638ea [id=57b47111637549ec8b103e1981a8691f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.417235 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0414dcb8a44148a4ab5d7f5b15e9592d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.420506 739 tablet_service.cc:1467] Processing CreateTablet for tablet 355a19649d1645e6a22773198ef8e49f (DEFAULT_TABLE table=default.Table_ac99de587ebb4dad9590199c16222c21 [id=25e8ec6fefa54b118b3ae7f8ae209a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.420797 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355a19649d1645e6a22773198ef8e49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.421166 750 tablet_service.cc:1467] Processing CreateTablet for tablet c299960dac724d1d9651626f1b579e36 (DEFAULT_TABLE table=default.Table_4a67fd289e384cc588d7d1ae3f0ed067 [id=f374feff4cd54ce2bcacf0142628ffc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.421416 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c299960dac724d1d9651626f1b579e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.423667 1164 tablet_bootstrap.cc:492] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.424746 738 tablet_service.cc:1467] Processing CreateTablet for tablet 324c810773f249db9e8325b09805141a (DEFAULT_TABLE table=default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.425038 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 324c810773f249db9e8325b09805141a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.427004 1164 tablet_bootstrap.cc:654] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.428016 876 tablet_service.cc:1467] Processing CreateTablet for tablet cbcd9fc1ed6d46209d8482c39ec6c3ba (DEFAULT_TABLE table=default.Table_05b6514d0e1a4ee2941c4f2b693bf29d [id=beb779f0483a410596b30bb5f1f0c8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.428519 754 tablet_service.cc:1467] Processing CreateTablet for tablet ae56f10005b145e19f55475075b578ae (DEFAULT_TABLE table=default.Table_c375b8b1b194457eb23f6594d4945a66 [id=96fbe2e92fa64520a7c1b813ad4b4744]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.428611 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbcd9fc1ed6d46209d8482c39ec6c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.428824 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae56f10005b145e19f55475075b578ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.429203 2469 raft_consensus.cc:491] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.430015 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4968ce56c6b4457097880a8ea7f924d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.432760 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 13a4837b72eb4008ad612a9fc86dfac7 (DEFAULT_TABLE table=default.Table_5b2238ba86fe41079543db9d13c4c1f4 [id=87305597cbdb42d595750887c01ea856]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.373158 744 tablet_service.cc:1467] Processing CreateTablet for tablet 44c70b9eba794f6a8ee393679db01a4c (DEFAULT_TABLE table=default.Table_4a7ea7c535504cfdb8ba930aa3f60055 [id=2a5d91dab0674283bd4338de14c96642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.433400 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c70b9eba794f6a8ee393679db01a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.434872 1016 tablet_service.cc:1467] Processing CreateTablet for tablet b7c6c8d5c58142c0af3e726ea57207b9 (DEFAULT_TABLE table=default.Table_10d8c601efc24a47b14b76858704b724 [id=d12776269f4045f78a988b1d8a56e691]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.437775 2340 raft_consensus.cc:491] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.437894 2340 raft_consensus.cc:513] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.438287 1843 raft_consensus.cc:491] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.438395 1843 raft_consensus.cc:513] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.438477 2340 leader_election.cc:290] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.438850 891 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2b1460d93e4c99afb45daf5e7d138e (DEFAULT_TABLE table=default.Table_e55f8e94ee304fd9ab86502d92eecf50 [id=239cde4385364164917f1181ee951874]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.438943 1843 leader_election.cc:290] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.439164 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a2b1460d93e4c99afb45daf5e7d138e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.439111 884 tablet_service.cc:1467] Processing CreateTablet for tablet 355a19649d1645e6a22773198ef8e49f (DEFAULT_TABLE table=default.Table_ac99de587ebb4dad9590199c16222c21 [id=25e8ec6fefa54b118b3ae7f8ae209a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.439222 1843 raft_consensus.cc:491] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.439306 1843 raft_consensus.cc:513] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.439604 885 tablet_service.cc:1467] Processing CreateTablet for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b (DEFAULT_TABLE table=default.Table_311231a4be29497898f249683c9fb37d [id=fcb55e82b323456aa31cb8d89a1b01b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.439663 887 tablet_service.cc:1467] Processing CreateTablet for tablet 17aa6950326d4226976e07c7cc986d39 (DEFAULT_TABLE table=default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.439756 893 tablet_service.cc:1467] Processing CreateTablet for tablet 2ebae7ff1b0b4ce6b558bf9a28b8fa72 (DEFAULT_TABLE table=default.Table_1d252b88a14246be9189c5c5261887c9 [id=7519c84d120b46959c1d6ec3b79d1a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.439832 1843 leader_election.cc:290] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.439412 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355a19649d1645e6a22773198ef8e49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.440047 883 tablet_service.cc:1467] Processing CreateTablet for tablet 324c810773f249db9e8325b09805141a (DEFAULT_TABLE table=default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.440130 878 tablet_service.cc:1467] Processing CreateTablet for tablet ae56f10005b145e19f55475075b578ae (DEFAULT_TABLE table=default.Table_c375b8b1b194457eb23f6594d4945a66 [id=96fbe2e92fa64520a7c1b813ad4b4744]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.440057 874 tablet_service.cc:1467] Processing CreateTablet for tablet 45204abca48c42ff82f6882532142be4 (DEFAULT_TABLE table=default.Table_a18f52945a28410fb91e6dfeef1cdcb9 [id=028275009f044112bd6b65996d609b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.440384 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 324c810773f249db9e8325b09805141a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.440317 889 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed31458aff04da0b1287c932a060ded (DEFAULT_TABLE table=default.Table_a0baa54d6f6f437d954be2ba0f6c03d9 [id=4e9ff162dda6488baf39463c68228a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.440580 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45204abca48c42ff82f6882532142be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.440130 879 tablet_service.cc:1467] Processing CreateTablet for tablet 7539a25fbf6c4e28b5538c8a29eaf529 (DEFAULT_TABLE table=default.Table_01bf4fd686f74e258977d847e2ee966e [id=f4f61992d591495f9e7a5a4b5c94fa32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.440822 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed31458aff04da0b1287c932a060ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.440590 892 tablet_service.cc:1467] Processing CreateTablet for tablet 237f3ec8c48a48bc975922f234fb3950 (DEFAULT_TABLE table=default.Table_c1102208539341c0a46b3222273132f3 [id=7934961e2c63425ca064f91c7073ed38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.440857 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe7404aa6ed49ca83d53d5d6e7d532c (DEFAULT_TABLE table=default.Table_f3626b709cb3478597936e4d350b38a1 [id=60785a1ed5fb4f54b29d5947a6f9f179]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.441051 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae56f10005b145e19f55475075b578ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.440620 881 tablet_service.cc:1467] Processing CreateTablet for tablet 0414dcb8a44148a4ab5d7f5b15e9592d (DEFAULT_TABLE table=default.Table_7accca692a1c423da80ab09d15b638ea [id=57b47111637549ec8b103e1981a8691f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.441226 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.441344 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0414dcb8a44148a4ab5d7f5b15e9592d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.441536 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7539a25fbf6c4e28b5538c8a29eaf529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.441751 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebae7ff1b0b4ce6b558bf9a28b8fa72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.441896 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17aa6950326d4226976e07c7cc986d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.444386 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 237f3ec8c48a48bc975922f234fb3950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.438577 882 tablet_service.cc:1467] Processing CreateTablet for tablet b7c6c8d5c58142c0af3e726ea57207b9 (DEFAULT_TABLE table=default.Table_10d8c601efc24a47b14b76858704b724 [id=d12776269f4045f78a988b1d8a56e691]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.455370 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c6c8d5c58142c0af3e726ea57207b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.466694 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.467280 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37f4c2ab9d248a1af817cfa2c2e2ff0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.467342 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2d04209fd34918b83978325a5e1df8" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:11.468034 848 leader_election.cc:343] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:11.469558 848 leader_election.cc:343] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:11.469911 848 leader_election.cc:343] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.432251 875 tablet_service.cc:1467] Processing CreateTablet for tablet 732708e96ddc477bbf9314cce3b78ddb (DEFAULT_TABLE table=default.Table_ac9b6dbf8a094c10aa1277694fc74c2a [id=c55656dbe6b342b7be8c366ad47b4f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.472817 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 732708e96ddc477bbf9314cce3b78ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.487814 1168 tablet_bootstrap.cc:492] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.487944 1168 ts_tablet_manager.cc:1397] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.219s user 0.005s sys 0.013s
I20250115 14:52:11.487977 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 234fd4e63b554cb583ef3df9dd64385c (DEFAULT_TABLE table=default.Table_f436acfa7f9d4abd8d6ca5f71c593faa [id=7fd95155ec424914992f1d2bf7f1453f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.488559 2516 raft_consensus.cc:491] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.488617 1168 raft_consensus.cc:357] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.488798 1168 raft_consensus.cc:383] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.488680 2516 raft_consensus.cc:513] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.488848 1168 raft_consensus.cc:738] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.489043 1168 consensus_queue.cc:260] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.489310 2516 leader_election.cc:290] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.489504 1168 ts_tablet_manager.cc:1428] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:11.489763 1168 tablet_bootstrap.cc:492] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.492209 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:11.492792 984 leader_election.cc:343] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.493295 876 tablet_service.cc:1467] Processing CreateTablet for tablet c299960dac724d1d9651626f1b579e36 (DEFAULT_TABLE table=default.Table_4a67fd289e384cc588d7d1ae3f0ed067 [id=f374feff4cd54ce2bcacf0142628ffc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.494134 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.494290 913 raft_consensus.cc:2463] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:11.495380 2469 raft_consensus.cc:513] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.496255 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 928e7cd1e2694d548bc973800f99796c (DEFAULT_TABLE table=default.Table_c7660a4693d04427a14694d941c8f631 [id=e28b2d17b9534b3cbf309ab00898ab9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.496899 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 4968ce56c6b4457097880a8ea7f924d8 (DEFAULT_TABLE table=default.Table_da2d5c7907c24047bd4f7c0f6b70aa45 [id=5ba88bc9fd1b40a881f2560c8271a49c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.497339 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cb3952b65cd477783c0425984bac152. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.497579 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 234fd4e63b554cb583ef3df9dd64385c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.497898 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 44c70b9eba794f6a8ee393679db01a4c (DEFAULT_TABLE table=default.Table_4a7ea7c535504cfdb8ba930aa3f60055 [id=2a5d91dab0674283bd4338de14c96642]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.498443 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fe7404aa6ed49ca83d53d5d6e7d532c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.498621 1018 tablet_service.cc:1467] Processing CreateTablet for tablet ace1012dde2b4ecbb9e58e1ad608644f (DEFAULT_TABLE table=default.Table_0ac18ecc16b2424881834e142ba358d9 [id=c76d044827ce4916b5f914622df4f9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.498786 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7c6c8d5c58142c0af3e726ea57207b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.498922 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13a4837b72eb4008ad612a9fc86dfac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.498953 1025 tablet_service.cc:1467] Processing CreateTablet for tablet cbcd9fc1ed6d46209d8482c39ec6c3ba (DEFAULT_TABLE table=default.Table_05b6514d0e1a4ee2941c4f2b693bf29d [id=beb779f0483a410596b30bb5f1f0c8d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.499219 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbcd9fc1ed6d46209d8482c39ec6c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.500016 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c70b9eba794f6a8ee393679db01a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.500406 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4968ce56c6b4457097880a8ea7f924d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.500551 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 928e7cd1e2694d548bc973800f99796c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.502219 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c299960dac724d1d9651626f1b579e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.502557 1843 raft_consensus.cc:491] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.502686 1843 raft_consensus.cc:513] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.503281 1843 leader_election.cc:290] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.505056 1164 tablet_bootstrap.cc:492] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.505225 1164 ts_tablet_manager.cc:1397] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20250115 14:52:11.506436 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 2a2b1460d93e4c99afb45daf5e7d138e (DEFAULT_TABLE table=default.Table_e55f8e94ee304fd9ab86502d92eecf50 [id=239cde4385364164917f1181ee951874]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.506522 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b (DEFAULT_TABLE table=default.Table_311231a4be29497898f249683c9fb37d [id=fcb55e82b323456aa31cb8d89a1b01b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.412253 1166 ts_tablet_manager.cc:1918] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.506830 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a2b1460d93e4c99afb45daf5e7d138e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.506892 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 355a19649d1645e6a22773198ef8e49f (DEFAULT_TABLE table=default.Table_ac99de587ebb4dad9590199c16222c21 [id=25e8ec6fefa54b118b3ae7f8ae209a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.507208 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ace1012dde2b4ecbb9e58e1ad608644f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.506844 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 2ebae7ff1b0b4ce6b558bf9a28b8fa72 (DEFAULT_TABLE table=default.Table_1d252b88a14246be9189c5c5261887c9 [id=7519c84d120b46959c1d6ec3b79d1a3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.508564 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 7539a25fbf6c4e28b5538c8a29eaf529 (DEFAULT_TABLE table=default.Table_01bf4fd686f74e258977d847e2ee966e [id=f4f61992d591495f9e7a5a4b5c94fa32]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.508885 983 leader_election.cc:304] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.509316 2516 raft_consensus.cc:2798] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:11.509404 2516 raft_consensus.cc:491] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:11.509475 2516 raft_consensus.cc:3054] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.512928 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 96692985a2c240099dbcb191fbd22384 (DEFAULT_TABLE table=default.Table_d100b9e08d3045d083f5800a8d651450 [id=c64973aed838415d9a339344fe777e15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.506497 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a11b7b0dd6494daf7b017edd4317a5" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.506223 1164 raft_consensus.cc:357] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.513419 1164 raft_consensus.cc:383] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.491067 1168 tablet_bootstrap.cc:654] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.513460 776 raft_consensus.cc:2463] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:11.514031 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.514253 848 leader_election.cc:304] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:11.513465 1164 raft_consensus.cc:738] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.514494 2341 raft_consensus.cc:2798] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:11.514570 2341 raft_consensus.cc:491] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:11.514636 2341 raft_consensus.cc:3054] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.514935 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aeca346a7746468dbbff053d933472" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:11.515388 984 leader_election.cc:343] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.515906 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aeca346a7746468dbbff053d933472" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.516075 913 raft_consensus.cc:2463] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
W20250115 14:52:11.516639 847 consensus_peers.cc:487] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 282d988bc41443c381bb0ba69b618f5d. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:11.516831 847 leader_election.cc:343] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 742792be08444792892bd9cd7336dda3
I20250115 14:52:11.516914 847 leader_election.cc:304] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.516963 2341 raft_consensus.cc:513] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:11.517319 847 leader_election.cc:343] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 1b2d04209fd34918b83978325a5e1df8
I20250115 14:52:11.517405 847 leader_election.cc:304] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.517520 2341 leader_election.cc:290] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.517647 2341 raft_consensus.cc:2743] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.517799 2341 raft_consensus.cc:2743] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.518051 1164 consensus_queue.cc:260] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.518257 1167 ts_tablet_manager.cc:1918] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.518237 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a11b7b0dd6494daf7b017edd4317a5" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:11.518404 776 raft_consensus.cc:3054] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.518432 1167 log.cc:1198] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/fa57c1fa96594a40b2551ad503127e4f
I20250115 14:52:11.518782 1167 ts_tablet_manager.cc:1939] T fa57c1fa96594a40b2551ad503127e4f P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.519100 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 940) took 2282 ms. Trace:
I20250115 14:52:11.506757 1166 log.cc:1198] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e5a059e2f0654a4aafe9e8082b8fde7a
I20250115 14:52:11.520579 1164 ts_tablet_manager.cc:1428] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250115 14:52:11.520660 1166 ts_tablet_manager.cc:1939] T e5a059e2f0654a4aafe9e8082b8fde7a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:11.520833 1164 tablet_bootstrap.cc:492] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.521337 776 raft_consensus.cc:2463] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:11.521471 1166 tablet_replica.cc:331] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:11.521634 1166 raft_consensus.cc:2238] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.521759 1166 raft_consensus.cc:2267] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.522426 1164 tablet_bootstrap.cc:654] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.525719 848 leader_election.cc:304] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:11.526087 2341 raft_consensus.cc:2798] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:11.526191 2341 raft_consensus.cc:695] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:11.526242 1167 rpcz_store.cc:276] 0115 14:52:09.236631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.236678 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:11.285317 (+2048639us) ts_tablet_manager.cc:1145] Deleting tablet fa57c1fa96594a40b2551ad503127e4f
0115 14:52:11.285328 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.477952 (+192624us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.518160 (+ 40208us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.519087 (+ 927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2048393}
I20250115 14:52:11.526404 2341 consensus_queue.cc:237] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.526767 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet fa57c1fa96594a40b2551ad503127e4f (table default.Table_958850c3f0a54c49a8dd42b180068897 [id=8242d7a08fcd4d4e836f6b70b1bacdc1]) successfully deleted
I20250115 14:52:11.527148 1167 tablet_replica.cc:331] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.527791 1167 raft_consensus.cc:2238] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.527998 1167 raft_consensus.cc:2267] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.528353 2273 raft_consensus.cc:491] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.529018 1167 ts_tablet_manager.cc:1905] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.530138 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:11.530879 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed31458aff04da0b1287c932a060ded (DEFAULT_TABLE table=default.Table_a0baa54d6f6f437d954be2ba0f6c03d9 [id=4e9ff162dda6488baf39463c68228a44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.531136 2469 leader_election.cc:290] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.530896 983 leader_election.cc:304] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.531653 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebae7ff1b0b4ce6b558bf9a28b8fa72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.532589 2273 raft_consensus.cc:513] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:11.533113 847 leader_election.cc:343] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: c37f4c2ab9d248a1af817cfa2c2e2ff0
I20250115 14:52:11.533218 847 leader_election.cc:304] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:11.533600 847 leader_election.cc:343] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b2a11b7b0dd6494daf7b017edd4317a5
I20250115 14:52:11.534232 1843 raft_consensus.cc:2743] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:11.535388 848 consensus_peers.cc:487] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 282d988bc41443c381bb0ba69b618f5d. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:11.537268 2469 raft_consensus.cc:2798] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:11.537753 1168 tablet_bootstrap.cc:492] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.537866 1168 ts_tablet_manager.cc:1397] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.000s
I20250115 14:52:11.539554 2273 leader_election.cc:290] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.540139 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246720dbb4ed4216ac87fb4a3df79e2c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.540714 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed31458aff04da0b1287c932a060ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.540961 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246720dbb4ed4216ac87fb4a3df79e2c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:11.541442 984 leader_election.cc:343] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.542562 1167 ts_tablet_manager.cc:1918] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.542644 1167 log.cc:1198] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/90c1602aef514dd29d5d2cffb043dbb6
I20250115 14:52:11.542894 1167 ts_tablet_manager.cc:1939] T 90c1602aef514dd29d5d2cffb043dbb6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.543187 2403 raft_consensus.cc:491] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.543270 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 938) took 2306 ms. Trace:
I20250115 14:52:11.543306 2403 raft_consensus.cc:513] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.543332 1167 rpcz_store.cc:276] 0115 14:52:09.236424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.237319 (+ 895us) service_pool.cc:225] Handling call
0115 14:52:11.527098 (+2289779us) ts_tablet_manager.cc:1145] Deleting tablet 90c1602aef514dd29d5d2cffb043dbb6
0115 14:52:11.527107 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.538489 (+ 11382us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.542483 (+ 3994us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.543258 (+ 775us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1792,"tablet-delete.queue_time_us":2289606}
I20250115 14:52:11.543753 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 90c1602aef514dd29d5d2cffb043dbb6 (table default.Table_950b48d1b1344479a7229041f044bbb6 [id=65eabb2be7d1479f889da99e50da551f]) successfully deleted
I20250115 14:52:11.543857 2403 leader_election.cc:290] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.544255 1167 tablet_replica.cc:331] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.544366 1168 raft_consensus.cc:357] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.544461 1167 raft_consensus.cc:2238] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.544529 1168 raft_consensus.cc:383] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.544574 1168 raft_consensus.cc:738] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.544605 1167 raft_consensus.cc:2267] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.544741 1168 consensus_queue.cc:260] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.544997 2469 raft_consensus.cc:491] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:11.545068 1167 ts_tablet_manager.cc:1905] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.545120 2469 raft_consensus.cc:3054] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.545311 1168 ts_tablet_manager.cc:1428] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250115 14:52:11.545535 1168 tablet_bootstrap.cc:492] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.546770 1168 tablet_bootstrap.cc:654] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.547425 2469 raft_consensus.cc:513] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.548031 1164 tablet_bootstrap.cc:492] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.548082 2469 leader_election.cc:290] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.548151 1164 ts_tablet_manager.cc:1397] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.001s
I20250115 14:52:11.548435 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aeca346a7746468dbbff053d933472" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:11.548767 983 leader_election.cc:343] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.548882 983 leader_election.cc:304] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.548908 1164 raft_consensus.cc:357] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:11.549026 984 leader_election.cc:343] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.549078 1164 raft_consensus.cc:383] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.549124 1164 raft_consensus.cc:738] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.549325 1164 consensus_queue.cc:260] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.549531 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aeca346a7746468dbbff053d933472" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:11.549693 913 raft_consensus.cc:3054] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.549937 1164 ts_tablet_manager.cc:1428] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:11.550326 1164 tablet_bootstrap.cc:492] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.551868 1164 tablet_bootstrap.cc:654] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.552098 913 raft_consensus.cc:2463] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:11.553335 1168 tablet_bootstrap.cc:492] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.553447 1168 ts_tablet_manager.cc:1397] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20250115 14:52:11.554033 1168 raft_consensus.cc:357] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.554199 1168 raft_consensus.cc:383] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.554252 1168 raft_consensus.cc:738] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.554409 1168 consensus_queue.cc:260] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.554947 1168 ts_tablet_manager.cc:1428] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250115 14:52:11.557865 1168 tablet_bootstrap.cc:492] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.559132 1168 tablet_bootstrap.cc:654] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.559511 1167 ts_tablet_manager.cc:1918] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.559659 1167 log.cc:1198] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/95718d39a7824517a36c49c3b86fef30
I20250115 14:52:11.559996 1167 ts_tablet_manager.cc:1939] T 95718d39a7824517a36c49c3b86fef30 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.568403 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:11.568840 2469 raft_consensus.cc:2743] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.569669 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7539a25fbf6c4e28b5538c8a29eaf529. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.568675 983 leader_election.cc:304] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.577530 1166 ts_tablet_manager.cc:1905] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.578723 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355a19649d1645e6a22773198ef8e49f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.581761 2516 raft_consensus.cc:513] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.582365 2469 raft_consensus.cc:2798] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:11.582880 1168 tablet_bootstrap.cc:492] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.583036 1168 ts_tablet_manager.cc:1397] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20250115 14:52:11.583328 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e5a059e2f0654a4aafe9e8082b8fde7a (table default.Table_dbb5dd7ec474409b9743a74e69cc1b44 [id=54db87fdb55743b8a485808e00abdbee]) successfully deleted
I20250115 14:52:11.583743 1168 raft_consensus.cc:357] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.583923 1168 raft_consensus.cc:383] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.583973 1168 raft_consensus.cc:738] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.584168 1168 consensus_queue.cc:260] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.585956 1164 tablet_bootstrap.cc:492] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.586146 1164 ts_tablet_manager.cc:1397] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.003s
I20250115 14:52:11.587028 1164 raft_consensus.cc:357] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.587288 1164 raft_consensus.cc:383] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.587410 1164 raft_consensus.cc:738] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.588528 1168 ts_tablet_manager.cc:1428] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:52:11.588445 1164 consensus_queue.cc:260] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.592936 1048 raft_consensus.cc:2388] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 73c86158cc2641f182ea50a06634592d in current term 1: Already voted for candidate c238c037a1c74725a001e2b4a1ff4304 in this term.
I20250115 14:52:11.593533 2469 raft_consensus.cc:695] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:11.593928 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:11.593281 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 939) took 2356 ms. Trace:
I20250115 14:52:11.594148 776 raft_consensus.cc:3054] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.594245 1164 ts_tablet_manager.cc:1428] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:52:11.594522 1164 tablet_bootstrap.cc:492] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.594290 1167 rpcz_store.cc:276] 0115 14:52:09.236552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.240343 (+ 3791us) service_pool.cc:225] Handling call
0115 14:52:11.544214 (+2303871us) ts_tablet_manager.cc:1145] Deleting tablet 95718d39a7824517a36c49c3b86fef30
0115 14:52:11.544223 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.556688 (+ 12465us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.559342 (+ 2654us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.592820 (+ 33478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2301873}
I20250115 14:52:11.596083 1164 tablet_bootstrap.cc:654] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:11.596060 847 leader_election.cc:343] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b2a11b7b0dd6494daf7b017edd4317a5
I20250115 14:52:11.596493 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.596670 913 raft_consensus.cc:2463] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:11.597692 710 leader_election.cc:304] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:11.598243 847 leader_election.cc:304] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.600109 2341 raft_consensus.cc:3054] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.600472 2495 raft_consensus.cc:491] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.600610 2495 raft_consensus.cc:513] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.601192 2495 leader_election.cc:290] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.601903 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377d711f052545ddbe499afe7ba3de28" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.602157 909 raft_consensus.cc:2463] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:11.602399 776 raft_consensus.cc:2463] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:11.602609 2341 raft_consensus.cc:2743] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.602598 710 leader_election.cc:304] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:11.602679 2403 raft_consensus.cc:2758] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250115 14:52:11.602907 2495 raft_consensus.cc:2798] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:11.603000 2495 raft_consensus.cc:491] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:11.603116 2495 raft_consensus.cc:3054] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.603766 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f06544890cbf460cbe218690e998eb1d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:11.605806 2495 raft_consensus.cc:513] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.606454 2495 leader_election.cc:290] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.607299 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "377d711f052545ddbe499afe7ba3de28" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:11.607453 909 raft_consensus.cc:3054] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.609910 909 raft_consensus.cc:2463] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:11.610203 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96692985a2c240099dbcb191fbd22384. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.611222 710 leader_election.cc:304] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:11.612088 2495 raft_consensus.cc:2798] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:11.612123 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 324c810773f249db9e8325b09805141a (DEFAULT_TABLE table=default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.612434 2495 raft_consensus.cc:695] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:11.612802 2495 consensus_queue.cc:237] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:11.614933 710 leader_election.cc:343] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 377d711f052545ddbe499afe7ba3de28
W20250115 14:52:11.615257 710 leader_election.cc:343] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 377d711f052545ddbe499afe7ba3de28
I20250115 14:52:11.616391 1168 tablet_bootstrap.cc:492] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.617723 1168 tablet_bootstrap.cc:654] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.621948 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 95718d39a7824517a36c49c3b86fef30 (table default.Table_5d8548b30284461086fdcd351454ad57 [id=d114031263ea4c87a6c4d12c9d6a0d32]) successfully deleted
I20250115 14:52:11.626973 984 leader_election.cc:304] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:11.627930 2516 leader_election.cc:290] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.626235 2469 consensus_queue.cc:237] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.629282 913 raft_consensus.cc:2463] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:11.631384 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 324c810773f249db9e8325b09805141a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.632660 2273 raft_consensus.cc:2798] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:11.639520 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 17aa6950326d4226976e07c7cc986d39 (DEFAULT_TABLE table=default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.641455 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17aa6950326d4226976e07c7cc986d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.649138 2273 raft_consensus.cc:695] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:11.638406 1167 tablet_replica.cc:331] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.665252 1167 raft_consensus.cc:2238] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.665432 1167 raft_consensus.cc:2267] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.665983 1167 ts_tablet_manager.cc:1905] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.671926 1167 ts_tablet_manager.cc:1918] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.672034 1167 log.cc:1198] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e5a059e2f0654a4aafe9e8082b8fde7a
I20250115 14:52:11.672281 2273 consensus_queue.cc:237] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:11.675428 847 consensus_peers.cc:487] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3b75d759f834d558c94cc981e338ac0. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:11.676502 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 45204abca48c42ff82f6882532142be4 (DEFAULT_TABLE table=default.Table_a18f52945a28410fb91e6dfeef1cdcb9 [id=028275009f044112bd6b65996d609b22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.674095 1024 tablet_service.cc:1467] Processing CreateTablet for tablet ae56f10005b145e19f55475075b578ae (DEFAULT_TABLE table=default.Table_c375b8b1b194457eb23f6594d4945a66 [id=96fbe2e92fa64520a7c1b813ad4b4744]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:11.682291 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 55089.4us
I20250115 14:52:11.682796 1167 ts_tablet_manager.cc:1939] T e5a059e2f0654a4aafe9e8082b8fde7a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.683624 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 941) took 2431 ms. Trace:
I20250115 14:52:11.684149 1167 rpcz_store.cc:276] 0115 14:52:09.252158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.252220 (+ 62us) service_pool.cc:225] Handling call
0115 14:52:11.618030 (+2365810us) ts_tablet_manager.cc:1145] Deleting tablet e5a059e2f0654a4aafe9e8082b8fde7a
0115 14:52:11.638345 (+ 20315us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.669093 (+ 30748us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.671840 (+ 2747us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.683604 (+ 11764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2365587}
I20250115 14:52:11.684948 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet e5a059e2f0654a4aafe9e8082b8fde7a (table default.Table_dbb5dd7ec474409b9743a74e69cc1b44 [id=54db87fdb55743b8a485808e00abdbee]) successfully deleted
I20250115 14:52:11.685312 1167 tablet_replica.cc:331] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.685549 1167 raft_consensus.cc:2238] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:11.685978 1167 raft_consensus.cc:2267] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.686767 1167 ts_tablet_manager.cc:1905] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.694358 1167 ts_tablet_manager.cc:1918] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:11.694993 1167 log.cc:1198] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/3f1e5d44b8064abbbf1b6419a1eca1ed
I20250115 14:52:11.695011 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 0414dcb8a44148a4ab5d7f5b15e9592d (DEFAULT_TABLE table=default.Table_7accca692a1c423da80ab09d15b638ea [id=57b47111637549ec8b103e1981a8691f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.695396 1167 ts_tablet_manager.cc:1939] T 3f1e5d44b8064abbbf1b6419a1eca1ed P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.695834 1164 tablet_bootstrap.cc:492] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.696125 1164 ts_tablet_manager.cc:1397] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.053s
I20250115 14:52:11.696250 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 943) took 2443 ms. Trace:
I20250115 14:52:11.696357 1167 rpcz_store.cc:276] 0115 14:52:09.252353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.252480 (+ 127us) service_pool.cc:225] Handling call
0115 14:52:11.685266 (+2432786us) ts_tablet_manager.cc:1145] Deleting tablet 3f1e5d44b8064abbbf1b6419a1eca1ed
0115 14:52:11.685276 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.691567 (+ 6291us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.694276 (+ 2709us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.696232 (+ 1956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2432606}
I20250115 14:52:11.696831 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 3f1e5d44b8064abbbf1b6419a1eca1ed (table default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91 [id=cdf45f410fda46f29cd6dbfd1cc45f28]) successfully deleted
I20250115 14:52:11.697197 1167 tablet_replica.cc:331] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.697417 1167 raft_consensus.cc:2238] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.697387 1164 raft_consensus.cc:357] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.697641 1167 raft_consensus.cc:2267] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.698169 1167 ts_tablet_manager.cc:1905] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.698681 1164 raft_consensus.cc:383] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.698906 1164 raft_consensus.cc:738] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.699441 1164 consensus_queue.cc:260] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.701156 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 237f3ec8c48a48bc975922f234fb3950 (DEFAULT_TABLE table=default.Table_c1102208539341c0a46b3222273132f3 [id=7934961e2c63425ca064f91c7073ed38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.701417 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45204abca48c42ff82f6882532142be4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.702076 1164 ts_tablet_manager.cc:1428] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:11.702818 1164 tablet_bootstrap.cc:492] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.704804 1164 tablet_bootstrap.cc:654] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.709924 1167 ts_tablet_manager.cc:1918] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:11.710044 1167 log.cc:1198] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/30f3a87bf8b6452d8afd8dfd836ee4e6
I20250115 14:52:11.710402 1167 ts_tablet_manager.cc:1939] T 30f3a87bf8b6452d8afd8dfd836ee4e6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.711155 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 944) took 2458 ms. Trace:
I20250115 14:52:11.711251 1167 rpcz_store.cc:276] 0115 14:52:09.252428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.252689 (+ 261us) service_pool.cc:225] Handling call
0115 14:52:11.697148 (+2444459us) ts_tablet_manager.cc:1145] Deleting tablet 30f3a87bf8b6452d8afd8dfd836ee4e6
0115 14:52:11.697158 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.700956 (+ 3798us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.709817 (+ 8861us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.711134 (+ 1317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2444329}
I20250115 14:52:11.711689 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 30f3a87bf8b6452d8afd8dfd836ee4e6 (table default.Table_062a9fcca27a4d90b73742167b635ae8 [id=5d05be5a21a44fe0b75c284bf3c83625]) successfully deleted
I20250115 14:52:11.712159 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 942) took 2459 ms. Trace:
I20250115 14:52:11.712284 1167 rpcz_store.cc:276] 0115 14:52:09.252261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.252843 (+ 582us) service_pool.cc:225] Handling call
0115 14:52:11.712027 (+2459184us) ts_tablet_manager.cc:1145] Deleting tablet e7315a41489a4f8296ad8b12c039b0a2
0115 14:52:11.712038 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.712142 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2459057}
W20250115 14:52:11.712879 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e7315a41489a4f8296ad8b12c039b0a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7315a41489a4f8296ad8b12c039b0a2 already in progress: creating tablet
I20250115 14:52:11.713317 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 946) took 2459 ms. Trace:
I20250115 14:52:11.713459 1167 rpcz_store.cc:276] 0115 14:52:09.253449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.254772 (+ 1323us) service_pool.cc:225] Handling call
0115 14:52:11.713198 (+2458426us) ts_tablet_manager.cc:1145] Deleting tablet e185963130bf49a1af88ab9e14e8e1bc
0115 14:52:11.713207 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.713299 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458299}
W20250115 14:52:11.713958 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e185963130bf49a1af88ab9e14e8e1bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e185963130bf49a1af88ab9e14e8e1bc already in progress: creating tablet
I20250115 14:52:11.714332 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 945) took 2461 ms. Trace:
I20250115 14:52:11.714439 1167 rpcz_store.cc:276] 0115 14:52:09.253340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.255873 (+ 2533us) service_pool.cc:225] Handling call
0115 14:52:11.714230 (+2458357us) ts_tablet_manager.cc:1145] Deleting tablet d6f2bc62621d402cacfcbe2eb74173f9
0115 14:52:11.714240 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.714315 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458192}
W20250115 14:52:11.714890 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d6f2bc62621d402cacfcbe2eb74173f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6f2bc62621d402cacfcbe2eb74173f9 already in progress: creating tablet
I20250115 14:52:11.715788 1167 tablet_replica.cc:331] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.716023 1167 raft_consensus.cc:2238] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.716213 1167 raft_consensus.cc:2267] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.716284 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae56f10005b145e19f55475075b578ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.716797 1167 ts_tablet_manager.cc:1905] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.722693 1167 ts_tablet_manager.cc:1918] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.722808 1167 log.cc:1198] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/83415f1c07ad495cabd287bdcd54dad4
I20250115 14:52:11.723091 1167 ts_tablet_manager.cc:1939] T 83415f1c07ad495cabd287bdcd54dad4 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.723829 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 947) took 2425 ms. Trace:
I20250115 14:52:11.723914 1167 rpcz_store.cc:276] 0115 14:52:09.298106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.298162 (+ 56us) service_pool.cc:225] Handling call
0115 14:52:11.715732 (+2417570us) ts_tablet_manager.cc:1145] Deleting tablet 83415f1c07ad495cabd287bdcd54dad4
0115 14:52:11.715741 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.719570 (+ 3829us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.722613 (+ 3043us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.723809 (+ 1196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2414433}
I20250115 14:52:11.724359 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 83415f1c07ad495cabd287bdcd54dad4 (table default.Table_458a573b4a4b4714950723b426f0d265 [id=1fd54f7dafdc4777bd06982391452889]) successfully deleted
I20250115 14:52:11.724786 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 961) took 2246 ms. Trace:
I20250115 14:52:11.724905 1167 rpcz_store.cc:276] 0115 14:52:09.478052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.478593 (+ 541us) service_pool.cc:225] Handling call
0115 14:52:11.724662 (+2246069us) ts_tablet_manager.cc:1145] Deleting tablet b3c31e24081d4a42aa933acbb80c6495
0115 14:52:11.724673 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.724771 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2245900}
W20250115 14:52:11.725402 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b3c31e24081d4a42aa933acbb80c6495 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3c31e24081d4a42aa933acbb80c6495 already in progress: creating tablet
I20250115 14:52:11.725793 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 963) took 2247 ms. Trace:
I20250115 14:52:11.725901 1167 rpcz_store.cc:276] 0115 14:52:09.478243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.478784 (+ 541us) service_pool.cc:225] Handling call
0115 14:52:11.725682 (+2246898us) ts_tablet_manager.cc:1145] Deleting tablet 0ee6906a03f546bbaaecea44cf0e7a68
0115 14:52:11.725691 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.725777 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2246737}
W20250115 14:52:11.726316 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0ee6906a03f546bbaaecea44cf0e7a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ee6906a03f546bbaaecea44cf0e7a68 already in progress: creating tablet
I20250115 14:52:11.726711 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 962) took 2248 ms. Trace:
I20250115 14:52:11.726830 1167 rpcz_store.cc:276] 0115 14:52:09.478158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.479120 (+ 962us) service_pool.cc:225] Handling call
0115 14:52:11.726611 (+2247491us) ts_tablet_manager.cc:1145] Deleting tablet 994fe2bae335497f8c07f50b91377074
0115 14:52:11.726620 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.726695 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2247318}
W20250115 14:52:11.727268 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 994fe2bae335497f8c07f50b91377074 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994fe2bae335497f8c07f50b91377074 already in progress: creating tablet
I20250115 14:52:11.727648 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 974) took 2037 ms. Trace:
I20250115 14:52:11.727783 1167 rpcz_store.cc:276] 0115 14:52:09.690401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.690459 (+ 58us) service_pool.cc:225] Handling call
0115 14:52:11.727554 (+2037095us) ts_tablet_manager.cc:1145] Deleting tablet 96369c35ca0d4a62b0676cac3dc4a813
0115 14:52:11.727563 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.727633 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035204}
W20250115 14:52:11.728214 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 96369c35ca0d4a62b0676cac3dc4a813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96369c35ca0d4a62b0676cac3dc4a813 already in progress: creating tablet
I20250115 14:52:11.728608 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 975) took 1941 ms. Trace:
I20250115 14:52:11.728716 1167 rpcz_store.cc:276] 0115 14:52:09.787223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.787274 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:11.728511 (+1941237us) ts_tablet_manager.cc:1145] Deleting tablet 7e600acd0ab74940a14f3022dbfa747f
0115 14:52:11.728520 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.728593 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1941064}
W20250115 14:52:11.729190 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7e600acd0ab74940a14f3022dbfa747f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e600acd0ab74940a14f3022dbfa747f already in progress: creating tablet
I20250115 14:52:11.729526 1167 tablet_replica.cc:331] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.729758 1167 raft_consensus.cc:2238] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:11.730098 1167 raft_consensus.cc:2267] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.731006 1167 ts_tablet_manager.cc:1905] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.731503 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 237f3ec8c48a48bc975922f234fb3950. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.733292 1026 tablet_service.cc:1467] Processing CreateTablet for tablet c299960dac724d1d9651626f1b579e36 (DEFAULT_TABLE table=default.Table_4a67fd289e384cc588d7d1ae3f0ed067 [id=f374feff4cd54ce2bcacf0142628ffc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:11.739754 1167 ts_tablet_manager.cc:1918] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:11.739866 1167 log.cc:1198] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/338c0f6f39fd4335bbcda733b28b8bf8
I20250115 14:52:11.739706 1162 tablet_bootstrap.cc:492] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:11.740197 1167 ts_tablet_manager.cc:1939] T 338c0f6f39fd4335bbcda733b28b8bf8 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.740926 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 976) took 1953 ms. Trace:
I20250115 14:52:11.741015 1167 rpcz_store.cc:276] 0115 14:52:09.787624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.787662 (+ 38us) service_pool.cc:225] Handling call
0115 14:52:11.729486 (+1941824us) ts_tablet_manager.cc:1145] Deleting tablet 338c0f6f39fd4335bbcda733b28b8bf8
0115 14:52:11.729495 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.733843 (+ 4348us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.739657 (+ 5814us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.740908 (+ 1251us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1941240}
I20250115 14:52:11.741420 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 338c0f6f39fd4335bbcda733b28b8bf8 (table default.Table_f367ac1a82704324a73f7079165a6e45 [id=6756cde43edd4d1e8a2768c101c4aa19]) successfully deleted
I20250115 14:52:11.741786 1167 tablet_replica.cc:331] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.742005 1167 raft_consensus.cc:2238] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.742197 1167 raft_consensus.cc:2267] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.742718 1167 ts_tablet_manager.cc:1905] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.748858 1167 ts_tablet_manager.cc:1918] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.749030 1167 log.cc:1198] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/07a000ab73dd46ac8e134784a156040a
I20250115 14:52:11.749387 1167 ts_tablet_manager.cc:1939] T 07a000ab73dd46ac8e134784a156040a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.750105 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 977) took 1962 ms. Trace:
I20250115 14:52:11.750242 1167 rpcz_store.cc:276] 0115 14:52:09.788059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.788117 (+ 58us) service_pool.cc:225] Handling call
0115 14:52:11.741727 (+1953610us) ts_tablet_manager.cc:1145] Deleting tablet 07a000ab73dd46ac8e134784a156040a
0115 14:52:11.741736 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.746065 (+ 4329us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.748780 (+ 2715us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.750087 (+ 1307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1953465}
I20250115 14:52:11.750783 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 07a000ab73dd46ac8e134784a156040a (table default.Table_9a090445edb041279a25064152cecee2 [id=3e116aeb5bcb4e098de4e46730e3dd96]) successfully deleted
I20250115 14:52:11.751137 1167 tablet_replica.cc:331] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.751358 1167 raft_consensus.cc:2238] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.751395 2516 raft_consensus.cc:491] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.751389 1046 raft_consensus.cc:1270] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:11.751549 1167 raft_consensus.cc:2267] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.751636 2516 raft_consensus.cc:513] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.752039 1167 ts_tablet_manager.cc:1905] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.752143 2546 consensus_queue.cc:1035] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:11.753216 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c299960dac724d1d9651626f1b579e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.758276 1167 ts_tablet_manager.cc:1918] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.758391 1167 log.cc:1198] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/4341977f363d40eeac71b7d33d1b43ac
I20250115 14:52:11.758685 1167 ts_tablet_manager.cc:1939] T 4341977f363d40eeac71b7d33d1b43ac P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.759464 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 978) took 1970 ms. Trace:
I20250115 14:52:11.759554 1167 rpcz_store.cc:276] 0115 14:52:09.788888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:09.788937 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:11.751091 (+1962154us) ts_tablet_manager.cc:1145] Deleting tablet 4341977f363d40eeac71b7d33d1b43ac
0115 14:52:11.751100 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.755183 (+ 4083us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.758195 (+ 3012us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.759442 (+ 1247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1962006}
I20250115 14:52:11.760021 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 4341977f363d40eeac71b7d33d1b43ac (table default.Table_ec25afcaa18b41b3906954d80092245d [id=b1e6efb09adf4c898362ed62f9fb98f9]) successfully deleted
I20250115 14:52:11.760478 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 983) took 1665 ms. Trace:
I20250115 14:52:11.760597 1167 rpcz_store.cc:276] 0115 14:52:10.095052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.095109 (+ 57us) service_pool.cc:225] Handling call
0115 14:52:11.760355 (+1665246us) ts_tablet_manager.cc:1145] Deleting tablet 0d7e33fc39d94c6286060f1e02f02372
0115 14:52:11.760364 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.760460 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665085}
W20250115 14:52:11.761111 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0d7e33fc39d94c6286060f1e02f02372 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d7e33fc39d94c6286060f1e02f02372 already in progress: creating tablet
I20250115 14:52:11.761507 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 984) took 1666 ms. Trace:
I20250115 14:52:11.761615 1167 rpcz_store.cc:276] 0115 14:52:10.095160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.095296 (+ 136us) service_pool.cc:225] Handling call
0115 14:52:11.761403 (+1666107us) ts_tablet_manager.cc:1145] Deleting tablet 0651be54e0b244a4bace9f47efc9b58a
0115 14:52:11.761412 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.761491 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1665984}
W20250115 14:52:11.762069 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0651be54e0b244a4bace9f47efc9b58a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0651be54e0b244a4bace9f47efc9b58a already in progress: creating tablet
I20250115 14:52:11.762378 2516 leader_election.cc:290] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.762434 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 985) took 1667 ms. Trace:
I20250115 14:52:11.762507 1167 rpcz_store.cc:276] 0115 14:52:10.096252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.097710 (+ 1458us) service_pool.cc:225] Handling call
0115 14:52:11.762338 (+1664628us) ts_tablet_manager.cc:1145] Deleting tablet 7ea8faf380974ad9888b7da3fab6bf3e
0115 14:52:11.762347 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.762419 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664464}
W20250115 14:52:11.762885 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7ea8faf380974ad9888b7da3fab6bf3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ea8faf380974ad9888b7da3fab6bf3e already in progress: creating tablet
I20250115 14:52:11.763252 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 986) took 1666 ms. Trace:
I20250115 14:52:11.763358 1167 rpcz_store.cc:276] 0115 14:52:10.096422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.098115 (+ 1693us) service_pool.cc:225] Handling call
0115 14:52:11.763157 (+1665042us) ts_tablet_manager.cc:1145] Deleting tablet dadeb51154c947209dc1e00ebf844b74
0115 14:52:11.763165 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.763235 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664903}
W20250115 14:52:11.763794 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet dadeb51154c947209dc1e00ebf844b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dadeb51154c947209dc1e00ebf844b74 already in progress: creating tablet
I20250115 14:52:11.764109 1167 tablet_replica.cc:331] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.764374 1167 raft_consensus.cc:2238] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.764679 1167 raft_consensus.cc:2267] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.765247 1167 ts_tablet_manager.cc:1905] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:11.768366 848 consensus_peers.cc:487] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:11.771203 1167 ts_tablet_manager.cc:1918] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250115 14:52:11.771374 1167 log.cc:1198] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/1839edd0d49b467d8bb31b3001ff89eb
I20250115 14:52:11.771767 1167 ts_tablet_manager.cc:1939] T 1839edd0d49b467d8bb31b3001ff89eb P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.772608 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 987) took 1620 ms. Trace:
I20250115 14:52:11.772735 1167 rpcz_store.cc:276] 0115 14:52:10.151720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.151895 (+ 175us) service_pool.cc:225] Handling call
0115 14:52:11.764065 (+1612170us) ts_tablet_manager.cc:1145] Deleting tablet 1839edd0d49b467d8bb31b3001ff89eb
0115 14:52:11.764074 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.768148 (+ 4074us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.771125 (+ 2977us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.772590 (+ 1465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1611981}
I20250115 14:52:11.773111 1167 tablet_replica.cc:331] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.773339 1167 raft_consensus.cc:2238] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:11.773638 1167 pending_rounds.cc:62] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:11.773728 1167 pending_rounds.cc:69] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558384138469376 op_type: NO_OP noop_request { }
I20250115 14:52:11.773892 1167 raft_consensus.cc:2883] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:11.774012 1167 raft_consensus.cc:2267] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.774646 1167 ts_tablet_manager.cc:1905] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.775987 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 1839edd0d49b467d8bb31b3001ff89eb (table default.Table_526b537d21ae4762b6881e46346414c6 [id=a724f6e4ad16485b8ee2445bb072be3e]) successfully deleted
I20250115 14:52:11.780441 1167 ts_tablet_manager.cc:1918] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:11.780544 1167 log.cc:1198] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/282d988bc41443c381bb0ba69b618f5d
I20250115 14:52:11.780865 1167 ts_tablet_manager.cc:1939] T 282d988bc41443c381bb0ba69b618f5d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.781656 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 989) took 1596 ms. Trace:
I20250115 14:52:11.781742 1167 rpcz_store.cc:276] 0115 14:52:10.184898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.184957 (+ 59us) service_pool.cc:225] Handling call
0115 14:52:11.773070 (+1588113us) ts_tablet_manager.cc:1145] Deleting tablet 282d988bc41443c381bb0ba69b618f5d
0115 14:52:11.773079 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.777609 (+ 4530us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.780362 (+ 2753us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.781637 (+ 1275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1587875}
I20250115 14:52:11.782048 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76d1bbf90c6416ca60805c9d8053bd4" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.782208 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 282d988bc41443c381bb0ba69b618f5d (table default.Table_4ab29505c6634887aa99118862eccf5d [id=83320921ba86467b9ad31fde698cb2d1]) successfully deleted
I20250115 14:52:11.782538 1167 tablet_replica.cc:331] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.782754 1167 raft_consensus.cc:2238] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.782915 1167 raft_consensus.cc:2267] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.783422 1167 ts_tablet_manager.cc:1905] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:11.787086 984 leader_election.cc:343] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.789830 1167 ts_tablet_manager.cc:1918] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.789932 1167 log.cc:1198] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/a600fa0f0ec049dca2803bf7d2c74a95
I20250115 14:52:11.790203 1167 ts_tablet_manager.cc:1939] T a600fa0f0ec049dca2803bf7d2c74a95 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.790871 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 988) took 1606 ms. Trace:
I20250115 14:52:11.790956 1167 rpcz_store.cc:276] 0115 14:52:10.184795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.185018 (+ 223us) service_pool.cc:225] Handling call
0115 14:52:11.782491 (+1597473us) ts_tablet_manager.cc:1145] Deleting tablet a600fa0f0ec049dca2803bf7d2c74a95
0115 14:52:11.782502 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.786991 (+ 4489us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.789728 (+ 2737us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.790852 (+ 1124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597116}
I20250115 14:52:11.791390 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet a600fa0f0ec049dca2803bf7d2c74a95 (table default.Table_b6a6d5ba30ed4c59b2b2b0a8dad0d93d [id=ab53ab1ed1b34dafa524ccb35291fd62]) successfully deleted
I20250115 14:52:11.791801 1167 tablet_replica.cc:331] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.792006 1167 raft_consensus.cc:2238] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.792166 1167 raft_consensus.cc:2267] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.792695 1167 ts_tablet_manager.cc:1905] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.794797 2546 raft_consensus.cc:491] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.794920 2546 raft_consensus.cc:513] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.795186 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b76d1bbf90c6416ca60805c9d8053bd4" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.795562 2546 leader_election.cc:290] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.796638 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 33c9f67c7bfd4ab19a7a8bf385c9596b
I20250115 14:52:11.796857 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 36ac2c9c80824c25bb974092b3aba744
W20250115 14:52:11.796844 847 leader_election.cc:343] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b5937d6ad0494734a402fdc594eb9415
I20250115 14:52:11.796983 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5937d6ad0494734a402fdc594eb9415" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:11.797485 848 leader_election.cc:343] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.797575 848 leader_election.cc:304] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.797570 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_29e435c06be34d20b3fe6ab2745835d8" } new_table_name: "default.Table_548ea14243474f25b73cfc9d56a806b3" modify_external_catalogs: true
I20250115 14:52:11.798207 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 36ac2c9c80824c25bb974092b3aba744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.798233 2546 raft_consensus.cc:2743] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.798097 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.799645 1167 ts_tablet_manager.cc:1918] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.799736 1167 log.cc:1198] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/377d711f052545ddbe499afe7ba3de28
I20250115 14:52:11.800040 1167 ts_tablet_manager.cc:1939] T 377d711f052545ddbe499afe7ba3de28 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.800853 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 991) took 1569 ms. Trace:
I20250115 14:52:11.800925 1167 rpcz_store.cc:276] 0115 14:52:10.231527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.231581 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:11.791756 (+1560175us) ts_tablet_manager.cc:1145] Deleting tablet 377d711f052545ddbe499afe7ba3de28
0115 14:52:11.791765 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.795467 (+ 3702us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.799563 (+ 4096us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.800835 (+ 1272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1559959}
I20250115 14:52:11.801313 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 377d711f052545ddbe499afe7ba3de28 (table default.Table_7ec0e7f2e4de4b14bd6cea8489658118 [id=898cecdad4254997a5bed59d46f7acec]) successfully deleted
I20250115 14:52:11.801652 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 992) took 1525 ms. Trace:
I20250115 14:52:11.801719 1167 rpcz_store.cc:276] 0115 14:52:10.276179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.276237 (+ 58us) service_pool.cc:225] Handling call
0115 14:52:11.801545 (+1525308us) ts_tablet_manager.cc:1145] Deleting tablet ca128bf3d6fc430298da932a78f54184
0115 14:52:11.801553 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.801637 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1524806}
W20250115 14:52:11.802131 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ca128bf3d6fc430298da932a78f54184 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca128bf3d6fc430298da932a78f54184 already in progress: creating tablet
I20250115 14:52:11.802439 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 993) took 1522 ms. Trace:
I20250115 14:52:11.802505 1167 rpcz_store.cc:276] 0115 14:52:10.279835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.279884 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:11.802344 (+1522460us) ts_tablet_manager.cc:1145] Deleting tablet 6fd405050a2548e1b9ea7d7bf34a35af
0115 14:52:11.802353 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.802425 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1522278}
I20250115 14:52:11.802502 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4a036dcd4a7a41fab5a082306bba837b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.802852 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6fd405050a2548e1b9ea7d7bf34a35af because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fd405050a2548e1b9ea7d7bf34a35af already in progress: creating tablet
W20250115 14:52:11.803069 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a036dcd4a7a41fab5a082306bba837b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.803107 1167 tablet_replica.cc:331] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.803272 1167 raft_consensus.cc:2238] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:11.803566 1167 raft_consensus.cc:2267] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.804065 1167 ts_tablet_manager.cc:1905] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.804904 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_86cbb9218b1e442f8537cf2dd2d329af" } new_table_name: "default.Table_81aa5a4246744ef69326175dae48e122" modify_external_catalogs: true
I20250115 14:52:11.805145 2403 raft_consensus.cc:491] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.805289 2403 raft_consensus.cc:513] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.805891 2403 leader_election.cc:290] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.805989 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a696017e859f474bb1fd062df44c224d
I20250115 14:52:11.806305 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aeca346a7746468dbbff053d933472" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:11.806514 913 raft_consensus.cc:2388] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d7ecece2ab49e1be7079216517d13b in current term 1: Already voted for candidate c238c037a1c74725a001e2b4a1ff4304 in this term.
I20250115 14:52:11.806597 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9e22bc322f6e4b8bbcbdb9e756b33180" } new_table_name: "default.Table_408fb30cc9914eebb963d1ef2094344f" modify_external_catalogs: true
I20250115 14:52:11.807011 875 tablet_service.cc:1514] Processing DeleteTablet for tablet a696017e859f474bb1fd062df44c224d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.808270 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c23b26e4b7ff4c699d1c86adf6731c71
W20250115 14:52:11.795637 983 leader_election.cc:343] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.808579 983 leader_election.cc:304] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.808951 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3c27ad12ca5f4ef3b72169e93eaf971f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:11.809170 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92aeca346a7746468dbbff053d933472" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:52:11.809432 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c27ad12ca5f4ef3b72169e93eaf971f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.809708 710 leader_election.cc:304] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:11.810149 2403 raft_consensus.cc:3054] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:11.811317 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fc5e4c7ec1f426dadbafe85d5eebdc9 consensus state has no leader
I20250115 14:52:11.811723 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4f994af897944da0b4e1f9cf4a90e137"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.812196 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f994af897944da0b4e1f9cf4a90e137 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.812861 875 tablet_service.cc:1514] Processing DeleteTablet for tablet c23b26e4b7ff4c699d1c86adf6731c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.813335 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.813531 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 377d711f052545ddbe499afe7ba3de28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.813722 752 tablet_service.cc:1514] Processing DeleteTablet for tablet a696017e859f474bb1fd062df44c224d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.814220 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 36ac2c9c80824c25bb974092b3aba744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.814400 744 tablet_service.cc:1514] Processing DeleteTablet for tablet c23b26e4b7ff4c699d1c86adf6731c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.815376 2403 raft_consensus.cc:2743] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.815919 2564 raft_consensus.cc:2743] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.820411 1167 ts_tablet_manager.cc:1918] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:11.820535 1167 log.cc:1198] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f3b75d759f834d558c94cc981e338ac0
I20250115 14:52:11.820906 1167 ts_tablet_manager.cc:1939] T f3b75d759f834d558c94cc981e338ac0 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.821673 2568 tablet.cc:1721] T 2484885df0da46f381d625a26e211c1e P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:11.821700 2567 tablet.cc:1721] T 2484885df0da46f381d625a26e211c1e P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:11.821985 455 catalog_manager.cc:5526] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:11.822422 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 92aeca346a7746468dbbff053d933472 (table default.Table_35a3371b3a6c4f35b2990f828671681e [id=60ff7f730d394990a0263e322d28d7ff]). Expected version 4 got 0
W20250115 14:52:11.822626 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23ad6f98f8fa445ab2100f29a7ffb991 consensus state has no leader
W20250115 14:52:11.823379 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b consensus state has no leader
I20250115 14:52:11.823772 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f8f4fd3823584b899089dc4b570b2756"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:11.806089 1162 ts_tablet_manager.cc:1397] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.532s user 0.008s sys 0.010s
W20250115 14:52:11.824319 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8f4fd3823584b899089dc4b570b2756 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.824956 1162 raft_consensus.cc:357] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.825124 1162 raft_consensus.cc:383] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.825165 1162 raft_consensus.cc:738] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.825378 1162 consensus_queue.cc:260] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.825572 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b2a9e7fffbe84c459c9b07db2c59dca0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:11.825862 1162 ts_tablet_manager.cc:1428] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:11.826092 1162 tablet_bootstrap.cc:492] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:11.826074 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2a9e7fffbe84c459c9b07db2c59dca0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.827545 1162 tablet_bootstrap.cc:654] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.833989 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 994) took 1517 ms. Trace:
I20250115 14:52:11.834098 1167 rpcz_store.cc:276] 0115 14:52:10.316835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.316886 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:11.803068 (+1486182us) ts_tablet_manager.cc:1145] Deleting tablet f3b75d759f834d558c94cc981e338ac0
0115 14:52:11.803076 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.807203 (+ 4127us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.820278 (+ 13075us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.833966 (+ 13688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1485988}
I20250115 14:52:11.834491 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1005) took 1197 ms. Trace:
I20250115 14:52:11.834578 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f3b75d759f834d558c94cc981e338ac0 (table default.Table_01bf4fd686f74e258977d847e2ee966e [id=f4f61992d591495f9e7a5a4b5c94fa32]) successfully deleted
I20250115 14:52:11.834580 1167 rpcz_store.cc:276] 0115 14:52:10.636981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.637462 (+ 481us) service_pool.cc:225] Handling call
0115 14:52:11.834365 (+1196903us) ts_tablet_manager.cc:1145] Deleting tablet 3963e79868284401a915cac5ef258155
0115 14:52:11.834376 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.834473 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196348}
W20250115 14:52:11.835005 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3963e79868284401a915cac5ef258155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3963e79868284401a915cac5ef258155 already in progress: creating tablet
W20250115 14:52:11.835621 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 570a78d2b40246ac8f460dcca569ec9d consensus state has no leader
I20250115 14:52:11.835959 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f047206807ce42eb89594d08a8209682" } modify_external_catalogs: true
I20250115 14:52:11.836134 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f047206807ce42eb89594d08a8209682" } modify_external_catalogs: true
I20250115 14:52:11.836577 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1004) took 1199 ms. Trace:
I20250115 14:52:11.836645 1167 rpcz_store.cc:276] 0115 14:52:10.636832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.638047 (+ 1215us) service_pool.cc:225] Handling call
0115 14:52:11.836454 (+1198407us) ts_tablet_manager.cc:1145] Deleting tablet 13053264b5ca41dd85e53d928aa511fd
0115 14:52:11.836466 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.836558 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1198289}
W20250115 14:52:11.837044 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 13053264b5ca41dd85e53d928aa511fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13053264b5ca41dd85e53d928aa511fd already in progress: creating tablet
I20250115 14:52:11.837365 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1003) took 1200 ms. Trace:
I20250115 14:52:11.837436 1167 rpcz_store.cc:276] 0115 14:52:10.636703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.640661 (+ 3958us) service_pool.cc:225] Handling call
0115 14:52:11.837267 (+1196606us) ts_tablet_manager.cc:1145] Deleting tablet 64a76d2772534687b1391e45b9cb984b
0115 14:52:11.837276 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.837350 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196453}
W20250115 14:52:11.837823 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 64a76d2772534687b1391e45b9cb984b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64a76d2772534687b1391e45b9cb984b already in progress: creating tablet
W20250115 14:52:11.837824 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a79f880415a4b16aabdbb7cd989efdb consensus state has no leader
I20250115 14:52:11.838265 447 master_service.cc:946] ReplaceTablet: received request to replace tablet c6cb8dfa7c42480bbd139d5acf059a57 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:11.838392 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_442bce8b3c9842cfba4b7f94c280eb3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.838900 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_442bce8b3c9842cfba4b7f94c280eb3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.839272 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1006) took 1182 ms. Trace:
I20250115 14:52:11.839340 1167 rpcz_store.cc:276] 0115 14:52:10.656989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.657056 (+ 67us) service_pool.cc:225] Handling call
0115 14:52:11.839179 (+1182123us) ts_tablet_manager.cc:1145] Deleting tablet 9727430a15574ec0b24a5a64da744cac
0115 14:52:11.839188 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.839256 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1181972}
W20250115 14:52:11.839715 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9727430a15574ec0b24a5a64da744cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9727430a15574ec0b24a5a64da744cac already in progress: creating tablet
I20250115 14:52:11.841266 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fa6bcdff5ed349439e0b9e997ed620cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:11.841743 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa6bcdff5ed349439e0b9e997ed620cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:11.844798 448 master_service.cc:946] ReplaceTablet: received request to replace tablet c574dba6b2a04915bdd871de4dff4df0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:11.847476 875 tablet_service.cc:1514] Processing DeleteTablet for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 with delete_type TABLET_DATA_DELETED (replaced by tablet bc1440e3cd444cf8b3365a0af8de8958) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.847666 876 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9b57b8b9434f55ace5a4f47380df54 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8bae9867d745afb1dfdf47bcb881da) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.847787 875 tablet_service.cc:1514] Processing DeleteTablet for tablet b5177265ba974ad6a381adb1e8c49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.847893 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2d04209fd34918b83978325a5e1df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.847996 875 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1ce59e05054a009cd36e12db35d449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.848105 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed with delete_type TABLET_DATA_DELETED (replaced by tablet 5b33c876fab1476aba3b1f6e799fa96c) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.848353 889 tablet_service.cc:1514] Processing DeleteTablet for tablet b5937d6ad0494734a402fdc594eb9415 with delete_type TABLET_DATA_DELETED (replaced by tablet f2ba4086a6bf4bfa83e740b31dfb53f8) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.848119 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 338c0f6f39fd4335bbcda733b28b8bf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.849102 448 master_service.cc:946] ReplaceTablet: received request to replace tablet 3be094dcd3cb4f9d9259d354300c1600 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:11.852682 1167 tablet_replica.cc:331] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.852934 1167 raft_consensus.cc:2238] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:11.853305 1167 raft_consensus.cc:2267] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.853927 1167 ts_tablet_manager.cc:1905] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.856951 446 master_service.cc:946] ReplaceTablet: received request to replace tablet a2fe51b51994491ab38a644bfc55e3ac from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:11.794734 1751 ts_tablet_manager.cc:1918] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.858337 1751 log.cc:1198] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a9f0ecfceecf4acdbf2c2fbec88ff3cb
I20250115 14:52:11.858664 1751 ts_tablet_manager.cc:1939] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:11.859390 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1010) took 1313 ms. Trace:
I20250115 14:52:11.859467 1751 rpcz_store.cc:276] 0115 14:52:10.545826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.546815 (+ 989us) service_pool.cc:225] Handling call
0115 14:52:11.324487 (+777672us) ts_tablet_manager.cc:1145] Deleting tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb
0115 14:52:11.324497 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.363056 (+ 38559us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.794652 (+431596us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.859369 (+ 64717us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":777545}
I20250115 14:52:11.859784 1751 tablet_replica.cc:331] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:11.859948 1751 raft_consensus.cc:2238] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.860065 1751 raft_consensus.cc:2267] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.860534 1751 ts_tablet_manager.cc:1905] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.865211 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet a696017e859f474bb1fd062df44c224d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.865428 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 36ac2c9c80824c25bb974092b3aba744 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.865226 455 catalog_manager.cc:5526] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:11.865588 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet c23b26e4b7ff4c699d1c86adf6731c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.865705 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.868033 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb (table default.Table_1c51a5aaced84915a1dba864731460fe [id=ec452771986240f4a7654505a30d3d9d]) successfully deleted
I20250115 14:52:11.872087 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c6cb8dfa7c42480bbd139d5acf059a57
I20250115 14:52:11.872905 447 catalog_manager.cc:6596] ReplaceTablet: tablet c6cb8dfa7c42480bbd139d5acf059a57 deleted and replaced by tablet 70167f772ef246f2b1b5fdb380d6f4da
I20250115 14:52:11.873242 2273 raft_consensus.cc:491] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.873389 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ad417f06081d41248dff6f1eed175f29
I20250115 14:52:11.873474 2273 raft_consensus.cc:513] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.874503 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325cd3fd8b4d447998822163e575e758" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.874924 744 tablet_service.cc:1514] Processing DeleteTablet for tablet c6cb8dfa7c42480bbd139d5acf059a57 with delete_type TABLET_DATA_DELETED (replaced by tablet 70167f772ef246f2b1b5fdb380d6f4da) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.875126 749 tablet_service.cc:1514] Processing DeleteTablet for tablet ad417f06081d41248dff6f1eed175f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:11.875458 984 leader_election.cc:343] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.876209 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325cd3fd8b4d447998822163e575e758" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:11.876691 983 leader_election.cc:343] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.876798 983 leader_election.cc:304] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.877101 1164 tablet_bootstrap.cc:492] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.877099 2563 raft_consensus.cc:2743] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.877254 1164 ts_tablet_manager.cc:1397] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.175s user 0.009s sys 0.001s
I20250115 14:52:11.877262 2273 leader_election.cc:290] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.877545 876 tablet_service.cc:1514] Processing DeleteTablet for tablet c6cb8dfa7c42480bbd139d5acf059a57 with delete_type TABLET_DATA_DELETED (replaced by tablet 70167f772ef246f2b1b5fdb380d6f4da) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.877975 1164 raft_consensus.cc:357] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.878190 1164 raft_consensus.cc:383] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.878263 1164 raft_consensus.cc:738] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.878469 1164 consensus_queue.cc:260] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.878978 1164 ts_tablet_manager.cc:1428] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:11.879314 1164 tablet_bootstrap.cc:492] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.881237 1164 tablet_bootstrap.cc:654] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.882071 1168 tablet_bootstrap.cc:492] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.882145 876 tablet_service.cc:1514] Processing DeleteTablet for tablet ad417f06081d41248dff6f1eed175f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.882205 1168 ts_tablet_manager.cc:1397] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.266s user 0.031s sys 0.044s
I20250115 14:52:11.882787 1168 raft_consensus.cc:357] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.882959 1168 raft_consensus.cc:383] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.883000 1168 raft_consensus.cc:738] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.883219 1168 consensus_queue.cc:260] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.885676 1168 ts_tablet_manager.cc:1428] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:11.886086 1168 tablet_bootstrap.cc:492] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.887255 2546 raft_consensus.cc:491] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.887362 2546 raft_consensus.cc:513] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.887504 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_8652d6d9ec084553b5183aa372c2554a [id=c29c4105b83d4df9b617997c0c9c5b14] alter complete (version 1)
I20250115 14:52:11.887902 2546 leader_election.cc:290] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.888509 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eee3d3dfec464baf10c238babb9eb1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:11.888598 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eee3d3dfec464baf10c238babb9eb1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.888702 1048 raft_consensus.cc:2463] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
W20250115 14:52:11.889110 848 leader_election.cc:343] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.889425 847 leader_election.cc:304] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.891184 1168 tablet_bootstrap.cc:654] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.893009 2546 raft_consensus.cc:2798] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:11.893090 2546 raft_consensus.cc:491] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:11.893157 2546 raft_consensus.cc:3054] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.893307 1166 ts_tablet_manager.cc:1918] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.893409 1166 log.cc:1198] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/4341977f363d40eeac71b7d33d1b43ac
I20250115 14:52:11.893671 1166 ts_tablet_manager.cc:1939] T 4341977f363d40eeac71b7d33d1b43ac P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:11.894587 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 4341977f363d40eeac71b7d33d1b43ac (table default.Table_ec25afcaa18b41b3906954d80092245d [id=b1e6efb09adf4c898362ed62f9fb98f9]) successfully deleted
I20250115 14:52:11.895589 2546 raft_consensus.cc:513] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:11.896257 441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.60:40227 (error 108) [suppressed 8 similar messages]
I20250115 14:52:11.896710 1168 tablet_bootstrap.cc:492] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.896893 1168 ts_tablet_manager.cc:1397] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20250115 14:52:11.897616 1168 raft_consensus.cc:357] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.897693 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eee3d3dfec464baf10c238babb9eb1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:11.897897 1048 raft_consensus.cc:3054] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:11.898470 1751 ts_tablet_manager.cc:1918] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.898566 1751 log.cc:1198] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a2fe51b51994491ab38a644bfc55e3ac
I20250115 14:52:11.898810 1751 ts_tablet_manager.cc:1939] T a2fe51b51994491ab38a644bfc55e3ac P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:11.899433 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1011) took 1353 ms. Trace:
I20250115 14:52:11.899503 1751 rpcz_store.cc:276] 0115 14:52:10.545927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.548309 (+ 2382us) service_pool.cc:225] Handling call
0115 14:52:11.859730 (+1311421us) ts_tablet_manager.cc:1145] Deleting tablet a2fe51b51994491ab38a644bfc55e3ac
0115 14:52:11.859739 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.869137 (+ 9398us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.898366 (+ 29229us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.899416 (+ 1050us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1311259}
I20250115 14:52:11.899789 1751 tablet_replica.cc:331] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:11.899947 1751 raft_consensus.cc:2238] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.900058 1751 raft_consensus.cc:2267] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:11.900327 984 connection.cc:537] client connection to 127.0.104.60:40227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250115 14:52:11.900710 1048 raft_consensus.cc:2463] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
W20250115 14:52:11.900930 711 connection.cc:537] client connection to 127.0.104.60:40227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250115 14:52:11.903224 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3be094dcd3cb4f9d9259d354300c1600
I20250115 14:52:11.903741 448 catalog_manager.cc:6596] ReplaceTablet: tablet 3be094dcd3cb4f9d9259d354300c1600 deleted and replaced by tablet b3daf6416ad847aeaa83087cc81c9239
W20250115 14:52:11.900303 848 connection.cc:537] client connection to 127.0.104.60:40227 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250115 14:52:11.904810 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be094dcd3cb4f9d9259d354300c1600 with delete_type TABLET_DATA_DELETED (replaced by tablet b3daf6416ad847aeaa83087cc81c9239) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.902485 847 leader_election.cc:304] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:11.906623 2549 raft_consensus.cc:2798] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:11.906746 2549 raft_consensus.cc:695] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:11.906992 2549 consensus_queue.cc:237] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.908489 2546 leader_election.cc:290] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.909075 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a2fe51b51994491ab38a644bfc55e3ac (table default.Table_9f7b9d2bd72741fbb3132278c8a77d1c [id=15f2a471b89449459d7dbf804b4e922c]) successfully deleted
I20250115 14:52:11.909598 2273 raft_consensus.cc:491] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.909804 2273 raft_consensus.cc:513] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.910919 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8da9bd5ec841ebbc61b0f8de5ace3b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:11.911451 983 leader_election.cc:343] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.911653 2273 leader_election.cc:290] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:11.911885 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be094dcd3cb4f9d9259d354300c1600 with delete_type TABLET_DATA_DELETED (replaced by tablet b3daf6416ad847aeaa83087cc81c9239) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.912124 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet ad417f06081d41248dff6f1eed175f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:11 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.912353 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet c6cb8dfa7c42480bbd139d5acf059a57 with delete_type TABLET_DATA_DELETED (replaced by tablet 70167f772ef246f2b1b5fdb380d6f4da) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.912438 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eee3d3dfec464baf10c238babb9eb1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:11.912845 1166 tablet_replica.cc:331] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:11.913049 1166 raft_consensus.cc:2238] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.913234 1166 raft_consensus.cc:2267] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.913920 1162 tablet_bootstrap.cc:492] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:11.914032 1164 tablet_bootstrap.cc:492] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.914018 1167 ts_tablet_manager.cc:1918] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:11.914114 1167 log.cc:1198] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/c6f9262e808b4589a21257ed3fb0363b
I20250115 14:52:11.914268 1164 ts_tablet_manager.cc:1397] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.005s
I20250115 14:52:11.914450 1167 ts_tablet_manager.cc:1939] T c6f9262e808b4589a21257ed3fb0363b P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.914579 1162 ts_tablet_manager.cc:1397] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.000s
I20250115 14:52:11.915206 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1007) took 1257 ms. Trace:
I20250115 14:52:11.915128 1164 raft_consensus.cc:357] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.915307 1164 raft_consensus.cc:383] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.915288 1167 rpcz_store.cc:276] 0115 14:52:10.658020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.658079 (+ 59us) service_pool.cc:225] Handling call
0115 14:52:11.852605 (+1194526us) ts_tablet_manager.cc:1145] Deleting tablet c6f9262e808b4589a21257ed3fb0363b
0115 14:52:11.852615 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.870858 (+ 18243us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.913911 (+ 43053us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.915189 (+ 1278us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1194376}
I20250115 14:52:11.915582 1167 tablet_replica.cc:331] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.915763 1167 raft_consensus.cc:2238] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.915900 1167 raft_consensus.cc:2267] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.915984 1162 raft_consensus.cc:357] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.916173 1164 raft_consensus.cc:738] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.916083 1166 ts_tablet_manager.cc:1905] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.916393 1167 ts_tablet_manager.cc:1905] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.916543 1162 raft_consensus.cc:383] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.916723 1162 raft_consensus.cc:738] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.917131 1162 consensus_queue.cc:260] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.920410 1164 consensus_queue.cc:260] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.921954 1164 ts_tablet_manager.cc:1428] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250115 14:52:11.922950 1162 ts_tablet_manager.cc:1428] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:52:11.922966 1164 tablet_bootstrap.cc:492] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.923411 1168 raft_consensus.cc:383] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.923486 1168 raft_consensus.cc:738] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.923673 1168 consensus_queue.cc:260] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.924214 1168 ts_tablet_manager.cc:1428] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20250115 14:52:11.924552 1162 tablet_bootstrap.cc:492] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:11.926210 1162 tablet_bootstrap.cc:654] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.928508 1168 tablet_bootstrap.cc:492] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:11.929044 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 288b0e455a854464940f65a2ab909d0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 288b0e455a854464940f65a2ab909d0b already in progress: creating tablet
W20250115 14:52:11.929363 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 07a000ab73dd46ac8e134784a156040a because the tablet was not found. No further retry: Not found: Tablet not found: 07a000ab73dd46ac8e134784a156040a
W20250115 14:52:11.929569 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 84844b5f50af408ab7dda73f7421cd9d because the tablet was not found. No further retry: Not found: Tablet not found: 84844b5f50af408ab7dda73f7421cd9d
I20250115 14:52:11.930039 1168 tablet_bootstrap.cc:654] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:11.932066 848 leader_election.cc:343] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.932245 2549 raft_consensus.cc:491] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:11.932991 2549 raft_consensus.cc:513] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.933189 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]) successfully deleted
I20250115 14:52:11.933665 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4475441e4a467a8d593b742d2a5b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:11.934309 847 leader_election.cc:343] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b5177265ba974ad6a381adb1e8c49589
I20250115 14:52:11.935070 2549 leader_election.cc:290] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:11.935879 1167 ts_tablet_manager.cc:1918] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:11.935976 1167 log.cc:1198] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/6942d102069d4c33b23fa2c0b29f94a6
I20250115 14:52:11.936249 1167 ts_tablet_manager.cc:1939] T 6942d102069d4c33b23fa2c0b29f94a6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:11.936779 452 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 92aeca346a7746468dbbff053d933472 (table default.Table_35a3371b3a6c4f35b2990f828671681e [id=60ff7f730d394990a0263e322d28d7ff]). Expected version 4 got 0
I20250115 14:52:11.937003 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1010) took 1254 ms. Trace:
I20250115 14:52:11.938292 1751 ts_tablet_manager.cc:1905] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.937093 1167 rpcz_store.cc:276] 0115 14:52:10.682562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.682609 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:11.915541 (+1232932us) ts_tablet_manager.cc:1145] Deleting tablet 6942d102069d4c33b23fa2c0b29f94a6
0115 14:52:11.915550 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.919194 (+ 3644us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.935777 (+ 16583us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:11.936985 (+ 1208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1232740}
I20250115 14:52:11.940150 1168 tablet_bootstrap.cc:492] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.940339 1168 ts_tablet_manager.cc:1397] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250115 14:52:11.941109 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0414dcb8a44148a4ab5d7f5b15e9592d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:11.941061 1168 raft_consensus.cc:357] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.941346 1168 raft_consensus.cc:383] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.941442 1168 raft_consensus.cc:738] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.941673 1168 consensus_queue.cc:260] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.943908 1168 ts_tablet_manager.cc:1428] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:52:11.944202 1168 tablet_bootstrap.cc:492] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.945741 1168 tablet_bootstrap.cc:654] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:11.948794 441 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b -> Peer 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227): Couldn't send request to peer 1cc2e23c560f4096be7b14006bdfc525. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:11.924283 1164 tablet_bootstrap.cc:654] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.952459 1162 tablet_bootstrap.cc:492] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:11.952606 1162 ts_tablet_manager.cc:1397] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.002s
I20250115 14:52:11.924957 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be094dcd3cb4f9d9259d354300c1600 with delete_type TABLET_DATA_DELETED (replaced by tablet b3daf6416ad847aeaa83087cc81c9239) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.930461 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8da9bd5ec841ebbc61b0f8de5ace3b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.934813 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5177265ba974ad6a381adb1e8c49589" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:11.949622 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 6942d102069d4c33b23fa2c0b29f94a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.950026 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 95718d39a7824517a36c49c3b86fef30 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:11.957471 1162 raft_consensus.cc:357] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.957733 1162 raft_consensus.cc:383] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.957829 1162 raft_consensus.cc:738] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.958155 1162 consensus_queue.cc:260] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.958770 1162 ts_tablet_manager.cc:1428] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20250115 14:52:11.960013 984 leader_election.cc:343] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.960325 984 leader_election.cc:304] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.960582 2273 raft_consensus.cc:2743] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.964252 1162 tablet_bootstrap.cc:492] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:11.963184 848 leader_election.cc:343] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:11.965608 848 leader_election.cc:304] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:11.965888 2546 raft_consensus.cc:2743] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:11.966042 1162 tablet_bootstrap.cc:654] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.966492 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 0651be54e0b244a4bace9f47efc9b58a with delete_type TABLET_DATA_DELETED (replaced by tablet 324c810773f249db9e8325b09805141a) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:11.971015 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 398ac8af619c427287b3964443173864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.975451 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a5e97279185541b992340c24ede01656 (table default.Table_a8391687956e4259a71808f267cb4ff0 [id=361ed20953ab4e79b1112ae76b95a423]). Expected version 3 got 0
I20250115 14:52:11.981105 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 18f23815196047628d562da7312b010f from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:11.982659 1167 tablet_replica.cc:331] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:11.982887 1167 raft_consensus.cc:2238] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.983052 1167 raft_consensus.cc:2267] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:11.983844 1167 ts_tablet_manager.cc:1905] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:11.984351 1164 tablet_bootstrap.cc:492] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:11.984535 1164 ts_tablet_manager.cc:1397] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20250115 14:52:11.985229 1164 raft_consensus.cc:357] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.985471 1164 raft_consensus.cc:383] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.985563 1164 raft_consensus.cc:738] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.985695 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 6942d102069d4c33b23fa2c0b29f94a6 (table default.Table_25a5a732ec814f24893d09a6f0b38d2c [id=fa886a7e8fcd43a096892c6435e13c29]) successfully deleted
I20250115 14:52:11.985812 1164 consensus_queue.cc:260] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.986552 1164 ts_tablet_manager.cc:1428] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:11.987172 1168 tablet_bootstrap.cc:492] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:11.987298 1168 ts_tablet_manager.cc:1397] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.002s
I20250115 14:52:11.987960 1168 raft_consensus.cc:357] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.988173 1168 raft_consensus.cc:383] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.988229 1168 raft_consensus.cc:738] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.988442 1168 consensus_queue.cc:260] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:11.988930 1168 ts_tablet_manager.cc:1428] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:11.989044 1162 tablet_bootstrap.cc:492] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:11.989157 1162 ts_tablet_manager.cc:1397] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20250115 14:52:11.989166 1168 tablet_bootstrap.cc:492] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:11.989806 1162 raft_consensus.cc:357] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.989969 1162 raft_consensus.cc:383] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:11.990010 1162 raft_consensus.cc:738] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:11.990214 1162 consensus_queue.cc:260] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:11.990700 1168 tablet_bootstrap.cc:654] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.992511 1164 tablet_bootstrap.cc:492] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:11.995433 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4c2707915e4e97ad6c66cb249c6d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6afbdd0b585e4e7e82067dce061924d0) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:11.996819 1164 tablet_bootstrap.cc:654] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:11.997198 1162 ts_tablet_manager.cc:1428] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250115 14:52:11.997637 1166 ts_tablet_manager.cc:1918] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:11.997718 1166 log.cc:1198] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/be2fd31f74a94787a5e8ccae3c5ef245
I20250115 14:52:11.997992 1166 ts_tablet_manager.cc:1939] T be2fd31f74a94787a5e8ccae3c5ef245 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:11.999367 1166 tablet_replica.cc:331] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:11.999529 1166 raft_consensus.cc:2238] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:11.999517 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet be2fd31f74a94787a5e8ccae3c5ef245 (table default.Table_8ae862ddf3f14e56bdde1d9ecd5002ec [id=724c015c8f3f4aa581ac6faa84187b06]) successfully deleted
I20250115 14:52:11.999641 1166 raft_consensus.cc:2267] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.000085 1166 ts_tablet_manager.cc:1905] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.003129 1162 tablet_bootstrap.cc:492] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.006280 1751 ts_tablet_manager.cc:1918] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.006467 1751 log.cc:1198] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/005a4cd0e9f14e76b01afa49d946ee32
I20250115 14:52:12.012446 1751 ts_tablet_manager.cc:1939] T 005a4cd0e9f14e76b01afa49d946ee32 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:12.012701 1162 tablet_bootstrap.cc:654] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.016387 1162 tablet_bootstrap.cc:492] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.016639 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ba297fd1bf8c47dabb7b1dab458cf8df" } modify_external_catalogs: true
I20250115 14:52:12.016825 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ba297fd1bf8c47dabb7b1dab458cf8df" } modify_external_catalogs: true
I20250115 14:52:12.017212 1168 tablet_bootstrap.cc:492] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.017330 1168 ts_tablet_manager.cc:1397] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.004s
I20250115 14:52:12.017308 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6c9c88e11c6d4580889dd2b45811a57f" } modify_external_catalogs: true
I20250115 14:52:12.017438 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6c9c88e11c6d4580889dd2b45811a57f" } modify_external_catalogs: true
I20250115 14:52:12.017931 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_95f70e728c46403fa5002e0f2a01b5b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.018051 1168 raft_consensus.cc:357] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.018229 1168 raft_consensus.cc:383] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:52:12.018471 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95f70e728c46403fa5002e0f2a01b5b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.019623 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_81803594c5c741619ae708256092f827" } modify_external_catalogs: true
I20250115 14:52:12.019766 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_81803594c5c741619ae708256092f827" } modify_external_catalogs: true
I20250115 14:52:12.020195 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1032bbe4a65c4fa1b238c728e1ab2fdf" } modify_external_catalogs: true
I20250115 14:52:12.020365 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1032bbe4a65c4fa1b238c728e1ab2fdf" } modify_external_catalogs: true
I20250115 14:52:12.023170 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0b05d8cc5e384fccb6f6949ae2c33c98" } new_table_name: "default.Table_1a9d1352e9f0419582e92a1faaf29288" modify_external_catalogs: true
I20250115 14:52:12.023743 1162 ts_tablet_manager.cc:1397] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.002s
I20250115 14:52:12.024578 1162 raft_consensus.cc:357] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.024767 1162 raft_consensus.cc:383] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.024808 1162 raft_consensus.cc:738] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.025010 1162 consensus_queue.cc:260] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.025490 1162 ts_tablet_manager.cc:1428] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250115 14:52:12.025728 1162 tablet_bootstrap.cc:492] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.027302 1162 tablet_bootstrap.cc:654] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.031237 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_47fb50f420d34a0f946869232d283ecc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.031736 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47fb50f420d34a0f946869232d283ecc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.032830 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b21a9aaeef5245b2813b04521d7a113b" } new_table_name: "default.Table_3003c0e277fb481684d9ff7def8aab42" modify_external_catalogs: true
W20250115 14:52:12.035512 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet db57f7c3a4c14cc19d3a0624e5b1817a consensus state has no leader
I20250115 14:52:12.037706 1162 tablet_bootstrap.cc:492] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.037824 1162 ts_tablet_manager.cc:1397] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20250115 14:52:12.038462 1162 raft_consensus.cc:357] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.038625 1162 raft_consensus.cc:383] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.038664 1162 raft_consensus.cc:738] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.039031 1162 consensus_queue.cc:260] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.039485 1162 ts_tablet_manager.cc:1428] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:12.039875 1162 tablet_bootstrap.cc:492] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.040602 1164 tablet_bootstrap.cc:492] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:12.041464 1164 ts_tablet_manager.cc:1397] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20250115 14:52:12.041711 1162 tablet_bootstrap.cc:654] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.042248 1164 raft_consensus.cc:357] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.042486 1164 raft_consensus.cc:383] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.042588 1164 raft_consensus.cc:738] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.042835 1164 consensus_queue.cc:260] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.044138 2546 raft_consensus.cc:491] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.044271 2546 raft_consensus.cc:513] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.044894 2546 leader_election.cc:290] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.045398 1164 ts_tablet_manager.cc:1428] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250115 14:52:12.045735 1164 tablet_bootstrap.cc:492] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:12.045830 847 leader_election.cc:343] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6ce498ffd3f04c5d8eb642739a838903
I20250115 14:52:12.047127 1164 tablet_bootstrap.cc:654] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.049302 2546 raft_consensus.cc:491] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.049427 2546 raft_consensus.cc:513] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.049948 2546 leader_election.cc:290] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.050652 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c34919648447fac14208ede0ddc01" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:12.050863 1048 raft_consensus.cc:2463] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:12.051360 847 leader_election.cc:304] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:12.053088 2546 raft_consensus.cc:2798] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:12.053169 2546 raft_consensus.cc:491] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:12.053222 2546 raft_consensus.cc:3054] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.054026 1162 tablet_bootstrap.cc:492] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.054159 1162 ts_tablet_manager.cc:1397] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20250115 14:52:12.054420 1168 raft_consensus.cc:738] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.054679 1168 consensus_queue.cc:260] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.054934 1162 raft_consensus.cc:357] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.055312 1162 raft_consensus.cc:383] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.055375 1162 raft_consensus.cc:738] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.055560 1162 consensus_queue.cc:260] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.056047 1162 ts_tablet_manager.cc:1428] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:12.056546 1162 tablet_bootstrap.cc:492] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.057020 1168 ts_tablet_manager.cc:1428] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.040s user 0.015s sys 0.003s
I20250115 14:52:12.057296 1168 tablet_bootstrap.cc:492] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.058317 1162 tablet_bootstrap.cc:654] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.058501 1168 tablet_bootstrap.cc:654] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.058544 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_02969c37d40f489482e5eba20a9e2a8c" } new_table_name: "default.Table_2dfbaa258dca4a26a74af636127225a0" modify_external_catalogs: true
W20250115 14:52:12.059859 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2abbe321d0fc421d9a45d4dccc74e161 consensus state has no leader
I20250115 14:52:12.060216 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_662b6dfe0baa4437babff0046306d1dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.062463 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_662b6dfe0baa4437babff0046306d1dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.069681 1168 tablet_bootstrap.cc:492] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.069833 1168 ts_tablet_manager.cc:1397] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20250115 14:52:12.069880 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce498ffd3f04c5d8eb642739a838903" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.070466 1168 raft_consensus.cc:357] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.070638 1168 raft_consensus.cc:383] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.070688 1168 raft_consensus.cc:738] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.070693 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c34919648447fac14208ede0ddc01" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.070886 1168 consensus_queue.cc:260] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.071478 1168 ts_tablet_manager.cc:1428] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:12.072019 1168 tablet_bootstrap.cc:492] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:12.072111 848 leader_election.cc:343] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.072219 848 leader_election.cc:304] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.072624 2549 raft_consensus.cc:2743] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:12.072811 848 leader_election.cc:343] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:12.073107 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 336bf703400c4fadbe9a13986ce1fd86 consensus state has no leader
I20250115 14:52:12.073472 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ea09a159f86444068a6a52f86e33ae66" } new_table_name: "default.Table_72e496f5859b463caaac692c42725308" modify_external_catalogs: true
I20250115 14:52:12.073658 1168 tablet_bootstrap.cc:654] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.076910 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0ee70516fe3c4a12983e1448718e7943" } new_table_name: "default.Table_19d4a8e242364fe4937cdbd66c57029d" modify_external_catalogs: true
I20250115 14:52:12.077447 1162 tablet_bootstrap.cc:492] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.077562 1162 ts_tablet_manager.cc:1397] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20250115 14:52:12.078801 1162 raft_consensus.cc:357] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.079476 1162 raft_consensus.cc:383] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.079725 1162 raft_consensus.cc:738] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.079690 2273 raft_consensus.cc:491] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.079880 2273 raft_consensus.cc:513] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.080444 2273 leader_election.cc:290] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.081022 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e97279185541b992340c24ede01656" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:12.081677 983 leader_election.cc:343] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.081986 1162 consensus_queue.cc:260] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.082640 1162 ts_tablet_manager.cc:1428] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250115 14:52:12.082937 1162 tablet_bootstrap.cc:492] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.083323 1168 tablet_bootstrap.cc:492] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.083437 1168 ts_tablet_manager.cc:1397] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250115 14:52:12.084072 1168 raft_consensus.cc:357] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.084251 1168 raft_consensus.cc:383] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.084338 1168 raft_consensus.cc:738] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.084455 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e97279185541b992340c24ede01656" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.084542 1168 consensus_queue.cc:260] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.085119 1168 ts_tablet_manager.cc:1428] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:12.085166 1162 tablet_bootstrap.cc:654] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.085371 1168 tablet_bootstrap.cc:492] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.086781 1168 tablet_bootstrap.cc:654] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.089311 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1012) took 1543 ms. Trace:
I20250115 14:52:12.089402 1751 rpcz_store.cc:276] 0115 14:52:10.546041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.548844 (+ 2803us) service_pool.cc:225] Handling call
0115 14:52:11.899736 (+1350892us) ts_tablet_manager.cc:1145] Deleting tablet 005a4cd0e9f14e76b01afa49d946ee32
0115 14:52:11.899745 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:11.955874 (+ 56129us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.006170 (+ 50296us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.089292 (+ 83122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1350750}
I20250115 14:52:12.089690 1751 tablet_replica.cc:331] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:12.089870 1751 raft_consensus.cc:2238] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.090000 1751 raft_consensus.cc:2267] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.090462 1751 ts_tablet_manager.cc:1905] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:12.091090 984 leader_election.cc:343] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.091265 984 leader_election.cc:304] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.092592 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 005a4cd0e9f14e76b01afa49d946ee32 (table default.Table_d611f6594f554cfe99c5aa5f2f7ea0e2 [id=cfab75a29f6c4e388b90e1814fa72fae]) successfully deleted
W20250115 14:52:12.092779 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ed0b0ccdcff423d9bb2b9a83511616d has no consensus state
I20250115 14:52:12.093181 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ff8ce351b9514d26933886794f108c09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.093812 2273 raft_consensus.cc:2743] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:12.093853 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff8ce351b9514d26933886794f108c09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.095885 2535 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef770dae727b42f5a9513b6f3d68b84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef770dae727b42f5a9513b6f3d68b84b" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35583 } } peers { permanent_uuid: "7d97cf5f1fbb4bc6883711020e2fa3f3" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 45653 } } peers { permanent_uuid: "1cc2e23c560f4096be7b14006bdfc525" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 40227 } } }
I20250115 14:52:12.096750 2535 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b [sys.catalog]: This master's current role is: LEADER
I20250115 14:52:12.098048 1168 tablet_bootstrap.cc:492] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
W20250115 14:52:12.098138 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 856533ed254a49b4a461ed1ad3eb405e consensus state has no leader
I20250115 14:52:12.098170 1168 ts_tablet_manager.cc:1397] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20250115 14:52:12.098472 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_02dcf1af493a4be5857c31c1c50836a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.098584 2549 raft_consensus.cc:491] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.098692 2549 raft_consensus.cc:513] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.098822 1168 raft_consensus.cc:357] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.098982 1168 raft_consensus.cc:383] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:52:12.098971 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02dcf1af493a4be5857c31c1c50836a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.099025 1168 raft_consensus.cc:738] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.099265 2549 leader_election.cc:290] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.099232 1168 consensus_queue.cc:260] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.099721 1168 ts_tablet_manager.cc:1428] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250115 14:52:12.099985 1168 tablet_bootstrap.cc:492] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:12.100222 847 leader_election.cc:343] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: b803d8e103a5423f85c75c9006809b17
I20250115 14:52:12.102128 1168 tablet_bootstrap.cc:654] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.104105 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b803d8e103a5423f85c75c9006809b17" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:12.108273 848 leader_election.cc:343] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.108466 848 leader_election.cc:304] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.109143 1162 tablet_bootstrap.cc:492] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.109288 1162 ts_tablet_manager.cc:1397] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.008s
I20250115 14:52:12.110056 1162 raft_consensus.cc:357] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.110250 1162 raft_consensus.cc:383] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.110296 1162 raft_consensus.cc:738] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.110492 1162 consensus_queue.cc:260] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.110929 1162 ts_tablet_manager.cc:1428] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:12.111191 1162 tablet_bootstrap.cc:492] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.112613 2549 raft_consensus.cc:2743] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.113720 1162 tablet_bootstrap.cc:654] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.113794 1168 tablet_bootstrap.cc:492] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.113900 1168 ts_tablet_manager.cc:1397] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20250115 14:52:12.114488 1168 raft_consensus.cc:357] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.114660 1168 raft_consensus.cc:383] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.114715 1168 raft_consensus.cc:738] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.114900 1168 consensus_queue.cc:260] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.117648 1168 ts_tablet_manager.cc:1428] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:12.117941 1168 tablet_bootstrap.cc:492] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.119429 1168 tablet_bootstrap.cc:654] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.122957 1166 ts_tablet_manager.cc:1918] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.123088 1166 log.cc:1198] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b2a11b7b0dd6494daf7b017edd4317a5
I20250115 14:52:12.123080 1751 ts_tablet_manager.cc:1918] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.123174 1751 log.cc:1198] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/248cadcdef8b4e99923cdbca39690685
I20250115 14:52:12.123353 1166 ts_tablet_manager.cc:1939] T b2a11b7b0dd6494daf7b017edd4317a5 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:12.123442 1751 ts_tablet_manager.cc:1939] T 248cadcdef8b4e99923cdbca39690685 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:12.124441 2403 raft_consensus.cc:491] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.124590 2403 raft_consensus.cc:513] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.125245 2403 leader_election.cc:290] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:12.128065 710 leader_election.cc:343] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6942d102069d4c33b23fa2c0b29f94a6
W20250115 14:52:12.128432 710 leader_election.cc:343] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 6942d102069d4c33b23fa2c0b29f94a6
I20250115 14:52:12.128527 710 leader_election.cc:304] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:12.129087 2546 raft_consensus.cc:513] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.129415 2403 raft_consensus.cc:2743] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.129730 2546 leader_election.cc:290] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.131551 1162 tablet_bootstrap.cc:492] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.131669 1162 ts_tablet_manager.cc:1397] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20250115 14:52:12.132390 1162 raft_consensus.cc:357] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.132552 1162 raft_consensus.cc:383] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.132591 1162 raft_consensus.cc:738] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.132802 1162 consensus_queue.cc:260] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.132921 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c34919648447fac14208ede0ddc01" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:12.133116 1048 raft_consensus.cc:3054] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.133344 1162 ts_tablet_manager.cc:1428] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:12.133571 1162 tablet_bootstrap.cc:492] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.134574 1168 tablet_bootstrap.cc:492] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.134682 1168 ts_tablet_manager.cc:1397] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20250115 14:52:12.135301 1168 raft_consensus.cc:357] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.135455 1168 raft_consensus.cc:383] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.135493 1168 raft_consensus.cc:738] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.135675 1168 consensus_queue.cc:260] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.136108 1168 ts_tablet_manager.cc:1428] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250115 14:52:12.136605 1168 tablet_bootstrap.cc:492] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.137737 2273 raft_consensus.cc:491] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.137849 2273 raft_consensus.cc:513] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.137915 2549 raft_consensus.cc:491] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.138057 2549 raft_consensus.cc:513] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.138377 1168 tablet_bootstrap.cc:654] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.138383 2273 leader_election.cc:290] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.138638 2549 leader_election.cc:290] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.139333 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1df21ab18f45a8adf9fedd22b2b8a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:12.139847 983 leader_election.cc:343] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.140442 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc363f70f3f44d309f69fac9356c6257" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:12.140597 1046 raft_consensus.cc:2463] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:12.140619 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c34919648447fac14208ede0ddc01" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:12.140985 847 leader_election.cc:304] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:12.140977 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc363f70f3f44d309f69fac9356c6257" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:12.141464 848 leader_election.cc:343] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:12.141724 848 leader_election.cc:343] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.141491 2549 raft_consensus.cc:2798] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:12.142163 2549 raft_consensus.cc:491] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:12.142241 2549 raft_consensus.cc:3054] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.144546 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1df21ab18f45a8adf9fedd22b2b8a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:12.145741 984 leader_election.cc:343] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.145859 984 leader_election.cc:304] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.146099 2273 raft_consensus.cc:2743] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.139845 1162 tablet_bootstrap.cc:654] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.151542 770 raft_consensus.cc:3054] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.153548 1162 tablet_bootstrap.cc:492] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.153667 1162 ts_tablet_manager.cc:1397] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20250115 14:52:12.154381 1162 raft_consensus.cc:357] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.154592 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1020) took 1499 ms. Trace:
I20250115 14:52:12.154685 1751 rpcz_store.cc:276] 0115 14:52:10.654979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.655071 (+ 92us) service_pool.cc:225] Handling call
0115 14:52:12.089648 (+1434577us) ts_tablet_manager.cc:1145] Deleting tablet 248cadcdef8b4e99923cdbca39690685
0115 14:52:12.089658 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.120068 (+ 30410us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.122987 (+ 2919us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.154572 (+ 31585us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1434396}
I20250115 14:52:12.155095 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1024) took 1499 ms. Trace:
I20250115 14:52:12.155158 1751 rpcz_store.cc:276] 0115 14:52:10.655305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.655352 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:12.154956 (+1499604us) ts_tablet_manager.cc:1145] Deleting tablet 3963e79868284401a915cac5ef258155
0115 14:52:12.154967 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.155080 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499462}
I20250115 14:52:12.155444 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1019) took 1500 ms. Trace:
I20250115 14:52:12.155498 1751 rpcz_store.cc:276] 0115 14:52:10.654854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.655357 (+ 503us) service_pool.cc:225] Handling call
0115 14:52:12.155361 (+1500004us) ts_tablet_manager.cc:1145] Deleting tablet 64a76d2772534687b1391e45b9cb984b
0115 14:52:12.155368 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.155432 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499825}
I20250115 14:52:12.155704 1751 tablet_replica.cc:331] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:12.155877 1751 raft_consensus.cc:2238] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.156002 1751 raft_consensus.cc:2267] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.156508 1751 ts_tablet_manager.cc:1905] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.157274 1166 tablet_replica.cc:331] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:12.157444 1166 raft_consensus.cc:2238] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:12.157642 1166 pending_rounds.cc:62] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:12.157666 1168 tablet_bootstrap.cc:492] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.157684 1166 pending_rounds.cc:69] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558388689342464 op_type: NO_OP noop_request { }
I20250115 14:52:12.157785 1166 raft_consensus.cc:2883] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:12.157794 1168 ts_tablet_manager.cc:1397] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.002s
I20250115 14:52:12.157857 1166 raft_consensus.cc:2267] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.158959 1168 raft_consensus.cc:357] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.159137 1168 raft_consensus.cc:383] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.159178 1168 raft_consensus.cc:738] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.159385 1168 consensus_queue.cc:260] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.159557 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 248cadcdef8b4e99923cdbca39690685 (table default.Table_b64c63a30a1c4f328986efc44022083b [id=5d393887a5e54536a4139752211d21a1]) successfully deleted
W20250115 14:52:12.159801 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 3963e79868284401a915cac5ef258155 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3963e79868284401a915cac5ef258155 already in progress: creating tablet
I20250115 14:52:12.159956 1168 ts_tablet_manager.cc:1428] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20250115 14:52:12.160003 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 64a76d2772534687b1391e45b9cb984b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64a76d2772534687b1391e45b9cb984b already in progress: creating tablet
I20250115 14:52:12.160184 1168 tablet_bootstrap.cc:492] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.164762 1168 tablet_bootstrap.cc:654] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.169150 913 raft_consensus.cc:1270] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:12.169734 2546 raft_consensus.cc:491] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.169862 2546 raft_consensus.cc:513] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.170403 2546 leader_election.cc:290] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.171337 2273 consensus_queue.cc:1035] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:52:12.171933 847 leader_election.cc:343] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 580742eb5dac4dba9995809890e0e315
I20250115 14:52:12.172755 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "580742eb5dac4dba9995809890e0e315" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:12.174530 848 leader_election.cc:343] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.174628 848 leader_election.cc:304] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.175034 2341 raft_consensus.cc:2743] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.178068 771 raft_consensus.cc:1270] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:12.178644 2273 consensus_queue.cc:1035] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:12.154592 1162 raft_consensus.cc:383] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.182838 1162 raft_consensus.cc:738] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.183081 1162 consensus_queue.cc:260] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.183640 1162 ts_tablet_manager.cc:1428] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20250115 14:52:12.186463 1162 tablet_bootstrap.cc:492] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.188135 1162 tablet_bootstrap.cc:654] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.189754 2549 raft_consensus.cc:513] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.190397 2549 leader_election.cc:290] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.190856 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc363f70f3f44d309f69fac9356c6257" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:12.191016 1046 raft_consensus.cc:3054] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.194110 2568 tablet.cc:1721] T 92aeca346a7746468dbbff053d933472 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250115 14:52:12.195758 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc363f70f3f44d309f69fac9356c6257" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:12.195902 1166 ts_tablet_manager.cc:1905] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.196209 2563 raft_consensus.cc:491] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.196359 2563 raft_consensus.cc:513] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:12.196542 848 leader_election.cc:343] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.196916 2563 leader_election.cc:290] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:12.197409 440 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 336bf703400c4fadbe9a13986ce1fd86 (table default.Table_2dfbaa258dca4a26a74af636127225a0 [id=f29f6193992746e5bc5a0b5ee6a1a8d9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 336bf703400c4fadbe9a13986ce1fd86 consensus state has no leader
I20250115 14:52:12.198081 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b599c6b8174959b8b66e90e9242057" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.198451 1048 raft_consensus.cc:2463] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
W20250115 14:52:12.199127 984 leader_election.cc:343] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.201715 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b599c6b8174959b8b66e90e9242057" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:12.201930 913 raft_consensus.cc:2463] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:12.202288 983 leader_election.cc:304] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.202694 2563 raft_consensus.cc:2798] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:12.202762 2563 raft_consensus.cc:491] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:12.202826 2563 raft_consensus.cc:3054] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:12.203819 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:45653, user_credentials={real_user=slave}} blocked reactor thread for 98603.3us
I20250115 14:52:12.204334 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b2a11b7b0dd6494daf7b017edd4317a5 (table default.Table_8e068b869e304185abaccf2d63685cd3 [id=5a50430da9af480282aa31475f196737]) successfully deleted
I20250115 14:52:12.204412 2567 tablet.cc:1721] T 92aeca346a7746468dbbff053d933472 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250115 14:52:12.204572 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c9f67c7bfd4ab19a7a8bf385c9596b already in progress: creating tablet
I20250115 14:52:12.204993 1167 ts_tablet_manager.cc:1918] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.205088 1167 log.cc:1198] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/5b85d63974ab418eb1921dd605fbf2b0
I20250115 14:52:12.205358 1167 ts_tablet_manager.cc:1939] T 5b85d63974ab418eb1921dd605fbf2b0 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:12.208554 847 leader_election.cc:304] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.210024 2549 raft_consensus.cc:2798] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:12.210237 2549 raft_consensus.cc:695] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:12.210461 2549 consensus_queue.cc:237] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.211267 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8f3885b7d49b4591b043b5d2520d5c6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.211794 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f3885b7d49b4591b043b5d2520d5c6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.214784 876 tablet_service.cc:1467] Processing CreateTablet for tablet 23318b1ad0774a76880def17a64ac059 (DEFAULT_TABLE table=default.Table_5fc99dfa3f504b2f9a09dcfc542dde73 [id=5deba539d66a4c109c4fcb740c752e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.215011 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 23318b1ad0774a76880def17a64ac059 (DEFAULT_TABLE table=default.Table_5fc99dfa3f504b2f9a09dcfc542dde73 [id=5deba539d66a4c109c4fcb740c752e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.215111 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_39f9c4e24e5a4e54bde4760f73b4ff65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.215552 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39f9c4e24e5a4e54bde4760f73b4ff65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.216718 451 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 2484885df0da46f381d625a26e211c1e (table default.Table_8652d6d9ec084553b5183aa372c2554a [id=c29c4105b83d4df9b617997c0c9c5b14]). Expected version 1 got 0
I20250115 14:52:12.217267 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 93aaea90b0924327807e7f594154b14e (DEFAULT_TABLE table=default.Table_32d4e94660334e308afef5563b749191 [id=d48b3d302c2c41d3bbadfc013e2f23da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.217672 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_da940b0d8b7d4e409e73adab5c24ba76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.217715 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3c12a0a7eb42c5888961103f7eb178 (DEFAULT_TABLE table=default.Table_f6cc9ac2a299480b990c0a3ddcf355fe [id=577598c258fd43739333692301b8b386]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:12.218079 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da940b0d8b7d4e409e73adab5c24ba76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.219249 452 catalog_manager.cc:5526] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:12.219534 889 tablet_service.cc:1467] Processing CreateTablet for tablet 93aaea90b0924327807e7f594154b14e (DEFAULT_TABLE table=default.Table_32d4e94660334e308afef5563b749191 [id=d48b3d302c2c41d3bbadfc013e2f23da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.220005 883 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3c12a0a7eb42c5888961103f7eb178 (DEFAULT_TABLE table=default.Table_f6cc9ac2a299480b990c0a3ddcf355fe [id=577598c258fd43739333692301b8b386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.221818 1751 ts_tablet_manager.cc:1918] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.221920 1751 log.cc:1198] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/5265d5eeac83481ca97801e3475886da
I20250115 14:52:12.222234 1751 ts_tablet_manager.cc:1939] T 5265d5eeac83481ca97801e3475886da P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:12.223105 2564 raft_consensus.cc:491] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.223223 2564 raft_consensus.cc:513] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.223837 2564 leader_election.cc:290] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.224320 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3c12a0a7eb42c5888961103f7eb178. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.224416 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded663ac1197454bbde3c4c2d526002d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:12.224907 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93aaea90b0924327807e7f594154b14e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:12.224907 983 leader_election.cc:343] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.225492 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23318b1ad0774a76880def17a64ac059. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.226451 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_702932472ce94b91b4535e0015a38be1" } modify_external_catalogs: true
I20250115 14:52:12.226609 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_702932472ce94b91b4535e0015a38be1" } modify_external_catalogs: true
I20250115 14:52:12.227133 2564 raft_consensus.cc:491] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.227501 2564 raft_consensus.cc:513] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.228109 2564 leader_election.cc:290] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.229362 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 7da4a558e1ea4bccb868ad5c489e96d5 (DEFAULT_TABLE table=default.Table_b71be8b255d34a7b9b1e3fd33dc4fb8c [id=242b872821f846c398bad0e1b9057275]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.229684 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da4a558e1ea4bccb868ad5c489e96d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.230748 2549 raft_consensus.cc:491] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.230890 2549 raft_consensus.cc:513] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.231551 2549 leader_election.cc:290] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.232445 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8bae9867d745afb1dfdf47bcb881da" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:12.232533 847 leader_election.cc:343] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 4f1ce59e05054a009cd36e12db35d449
I20250115 14:52:12.233582 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 83415f1c07ad495cabd287bdcd54dad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.233968 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7da4a558e1ea4bccb868ad5c489e96d5 (DEFAULT_TABLE table=default.Table_b71be8b255d34a7b9b1e3fd33dc4fb8c [id=242b872821f846c398bad0e1b9057275]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.234229 752 tablet_service.cc:1467] Processing CreateTablet for tablet 23318b1ad0774a76880def17a64ac059 (DEFAULT_TABLE table=default.Table_5fc99dfa3f504b2f9a09dcfc542dde73 [id=5deba539d66a4c109c4fcb740c752e7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.234421 740 tablet_service.cc:1467] Processing CreateTablet for tablet c0567f95a3444fedb002e550615a99fa (DEFAULT_TABLE table=default.Table_edbe5b64c3fb4732a52d6af29150486e [id=6cb329da4f4a45948fab32b7bf1647b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.234611 744 tablet_service.cc:1467] Processing CreateTablet for tablet 93aaea90b0924327807e7f594154b14e (DEFAULT_TABLE table=default.Table_32d4e94660334e308afef5563b749191 [id=d48b3d302c2c41d3bbadfc013e2f23da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.234977 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4f3c12a0a7eb42c5888961103f7eb178 (DEFAULT_TABLE table=default.Table_f6cc9ac2a299480b990c0a3ddcf355fe [id=577598c258fd43739333692301b8b386]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.235571 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded663ac1197454bbde3c4c2d526002d" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.235939 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8bae9867d745afb1dfdf47bcb881da" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:12.236444 983 leader_election.cc:343] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.237251 875 tablet_service.cc:1467] Processing CreateTablet for tablet 7da4a558e1ea4bccb868ad5c489e96d5 (DEFAULT_TABLE table=default.Table_b71be8b255d34a7b9b1e3fd33dc4fb8c [id=242b872821f846c398bad0e1b9057275]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.237382 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1ce59e05054a009cd36e12db35d449" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.237720 880 tablet_service.cc:1467] Processing CreateTablet for tablet c0567f95a3444fedb002e550615a99fa (DEFAULT_TABLE table=default.Table_edbe5b64c3fb4732a52d6af29150486e [id=6cb329da4f4a45948fab32b7bf1647b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.238236 1016 tablet_service.cc:1467] Processing CreateTablet for tablet c0567f95a3444fedb002e550615a99fa (DEFAULT_TABLE table=default.Table_edbe5b64c3fb4732a52d6af29150486e [id=6cb329da4f4a45948fab32b7bf1647b1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:12.238559 848 leader_election.cc:343] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.238652 848 leader_election.cc:304] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.239141 2549 raft_consensus.cc:2743] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:12.239629 984 leader_election.cc:343] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.239732 984 leader_election.cc:304] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:12.240104 984 leader_election.cc:343] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.240195 984 leader_election.cc:304] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.240203 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0fe88a1365954ef5beb1d67b1cc4b747
I20250115 14:52:12.240702 2469 raft_consensus.cc:2743] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.240868 2469 raft_consensus.cc:2743] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.241779 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe88a1365954ef5beb1d67b1cc4b747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.242272 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe88a1365954ef5beb1d67b1cc4b747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.242614 881 tablet_service.cc:1467] Processing CreateTablet for tablet addb4972a8de427d99b0cbb646796c63 (DEFAULT_TABLE table=default.Table_030ac1279f7943c8bb81484f8da9a636 [id=66e44986a7094a8bbd53082afc1d58f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.242837 2604 mvcc.cc:204] Tried to move back new op lower bound from 7114558389236703232 to 7114558385990488064. Current Snapshot: MvccSnapshot[applied={T|T < 7114558389236703232}]
I20250115 14:52:12.243544 1022 tablet_service.cc:1467] Processing CreateTablet for tablet addb4972a8de427d99b0cbb646796c63 (DEFAULT_TABLE table=default.Table_030ac1279f7943c8bb81484f8da9a636 [id=66e44986a7094a8bbd53082afc1d58f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.243558 2610 tablet.cc:1721] T 2484885df0da46f381d625a26e211c1e P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:12.244789 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 18f23815196047628d562da7312b010f
I20250115 14:52:12.245210 455 catalog_manager.cc:6596] ReplaceTablet: tablet 18f23815196047628d562da7312b010f deleted and replaced by tablet 05bb99f36ba649d8b607e9a95ab58ed0
I20250115 14:52:12.246295 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f23815196047628d562da7312b010f with delete_type TABLET_DATA_DELETED (replaced by tablet 05bb99f36ba649d8b607e9a95ab58ed0) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.246417 882 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed0b0ccdcff423d9bb2b9a83511616d (DEFAULT_TABLE table=default.Table_72e496f5859b463caaac692c42725308 [id=c25247739e93462ca2f16a3076a614cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.247056 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f23815196047628d562da7312b010f with delete_type TABLET_DATA_DELETED (replaced by tablet 05bb99f36ba649d8b607e9a95ab58ed0) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.247537 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed0b0ccdcff423d9bb2b9a83511616d (DEFAULT_TABLE table=default.Table_72e496f5859b463caaac692c42725308 [id=c25247739e93462ca2f16a3076a614cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.249790 874 tablet_service.cc:1467] Processing CreateTablet for tablet 0163895f3b274712ba69890f7e49ab9d (DEFAULT_TABLE table=default.Table_35d4a8cd197f4504966cf56cc33f3a99 [id=77a2c3a379ea4838aef9d42489e31652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.250568 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 0163895f3b274712ba69890f7e49ab9d (DEFAULT_TABLE table=default.Table_35d4a8cd197f4504966cf56cc33f3a99 [id=77a2c3a379ea4838aef9d42489e31652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.251574 756 tablet_service.cc:1467] Processing CreateTablet for tablet addb4972a8de427d99b0cbb646796c63 (DEFAULT_TABLE table=default.Table_030ac1279f7943c8bb81484f8da9a636 [id=66e44986a7094a8bbd53082afc1d58f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.251662 745 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f23815196047628d562da7312b010f with delete_type TABLET_DATA_DELETED (replaced by tablet 05bb99f36ba649d8b607e9a95ab58ed0) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.251895 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe88a1365954ef5beb1d67b1cc4b747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.252192 739 tablet_service.cc:1467] Processing CreateTablet for tablet 0163895f3b274712ba69890f7e49ab9d (DEFAULT_TABLE table=default.Table_35d4a8cd197f4504966cf56cc33f3a99 [id=77a2c3a379ea4838aef9d42489e31652]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.251575 755 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed0b0ccdcff423d9bb2b9a83511616d (DEFAULT_TABLE table=default.Table_72e496f5859b463caaac692c42725308 [id=c25247739e93462ca2f16a3076a614cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.253614 1162 tablet_bootstrap.cc:492] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.254215 748 tablet_service.cc:1467] Processing CreateTablet for tablet a5018e358d234ea28023da76a8c61a86 (DEFAULT_TABLE table=default.Table_08ec29c2d0ca4d7db6cd48a97784dced [id=f656e4d2688b41e9973bb3d06e010166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.254629 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23318b1ad0774a76880def17a64ac059. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.254621 1162 ts_tablet_manager.cc:1397] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.004s
I20250115 14:52:12.255463 1162 raft_consensus.cc:357] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.255703 1164 tablet_bootstrap.cc:492] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:12.255757 1162 raft_consensus.cc:383] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.255864 1162 raft_consensus.cc:738] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.256165 1162 consensus_queue.cc:260] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.256793 1162 ts_tablet_manager.cc:1428] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:12.256968 879 tablet_service.cc:1467] Processing CreateTablet for tablet a5018e358d234ea28023da76a8c61a86 (DEFAULT_TABLE table=default.Table_08ec29c2d0ca4d7db6cd48a97784dced [id=f656e4d2688b41e9973bb3d06e010166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.257087 1162 tablet_bootstrap.cc:492] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.257807 887 tablet_service.cc:1467] Processing CreateTablet for tablet ec9e7c8e59aa4ea59bad35c8a7591a79 (DEFAULT_TABLE table=default.Table_cc773ad5c57c4058baad0482e8dd2b79 [id=bd98be3208a44386865cd46afa869cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.258479 1029 tablet_service.cc:1467] Processing CreateTablet for tablet a5018e358d234ea28023da76a8c61a86 (DEFAULT_TABLE table=default.Table_08ec29c2d0ca4d7db6cd48a97784dced [id=f656e4d2688b41e9973bb3d06e010166]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.258922 1012 tablet_service.cc:1467] Processing CreateTablet for tablet ec9e7c8e59aa4ea59bad35c8a7591a79 (DEFAULT_TABLE table=default.Table_cc773ad5c57c4058baad0482e8dd2b79 [id=bd98be3208a44386865cd46afa869cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.259066 878 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc659e2d2d943a99aef530e7d71df88 (DEFAULT_TABLE table=default.Table_c046302be54a4e91a532c737044d0f87 [id=3c4da8cf0f3545f2be69be9d7802e19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.259413 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc659e2d2d943a99aef530e7d71df88 (DEFAULT_TABLE table=default.Table_c046302be54a4e91a532c737044d0f87 [id=3c4da8cf0f3545f2be69be9d7802e19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.259472 884 tablet_service.cc:1467] Processing CreateTablet for tablet d27a88650b024af19938618d9ed0cc23 (DEFAULT_TABLE table=default.Table_d02b19d9fa214584a8c9a02041a6da29 [id=90f226b870294f69a681feb108c87d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.259701 1164 ts_tablet_manager.cc:1397] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.214s user 0.013s sys 0.004s
I20250115 14:52:12.259894 1020 tablet_service.cc:1467] Processing CreateTablet for tablet d27a88650b024af19938618d9ed0cc23 (DEFAULT_TABLE table=default.Table_d02b19d9fa214584a8c9a02041a6da29 [id=90f226b870294f69a681feb108c87d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.259987 2341 raft_consensus.cc:491] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.260090 2341 raft_consensus.cc:513] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.260198 892 tablet_service.cc:1467] Processing CreateTablet for tablet c8e33ebec8b64f2398a7f1003b15cee7 (DEFAULT_TABLE table=default.Table_89373c553f9c407a9e205bfe0d1acf99 [id=26c53728ab22430b95ca322543118a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.260633 753 tablet_service.cc:1467] Processing CreateTablet for tablet d27a88650b024af19938618d9ed0cc23 (DEFAULT_TABLE table=default.Table_d02b19d9fa214584a8c9a02041a6da29 [id=90f226b870294f69a681feb108c87d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.260633 745 tablet_service.cc:1467] Processing CreateTablet for tablet ec9e7c8e59aa4ea59bad35c8a7591a79 (DEFAULT_TABLE table=default.Table_cc773ad5c57c4058baad0482e8dd2b79 [id=bd98be3208a44386865cd46afa869cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.261075 747 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc659e2d2d943a99aef530e7d71df88 (DEFAULT_TABLE table=default.Table_c046302be54a4e91a532c737044d0f87 [id=3c4da8cf0f3545f2be69be9d7802e19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.261456 737 tablet_service.cc:1467] Processing CreateTablet for tablet c8e33ebec8b64f2398a7f1003b15cee7 (DEFAULT_TABLE table=default.Table_89373c553f9c407a9e205bfe0d1acf99 [id=26c53728ab22430b95ca322543118a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.261960 1162 tablet_bootstrap.cc:654] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.262068 1164 raft_consensus.cc:357] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.262888 2341 leader_election.cc:290] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:12.263587 847 leader_election.cc:343] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 3b82924e735846b2914520684d1b8fc7
I20250115 14:52:12.263226 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b82924e735846b2914520684d1b8fc7" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.264369 1164 raft_consensus.cc:383] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.264485 1164 raft_consensus.cc:738] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
W20250115 14:52:12.264747 848 leader_election.cc:343] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.264848 848 leader_election.cc:304] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.265065 2341 raft_consensus.cc:2743] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.264747 1164 consensus_queue.cc:260] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.266983 1164 ts_tablet_manager.cc:1428] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250115 14:52:12.267778 1164 tablet_bootstrap.cc:492] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:12.268514 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1009) took 1586 ms. Trace:
I20250115 14:52:12.268643 1167 rpcz_store.cc:276] 0115 14:52:10.682402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.684420 (+ 2018us) service_pool.cc:225] Handling call
0115 14:52:11.982595 (+1298175us) ts_tablet_manager.cc:1145] Deleting tablet 5b85d63974ab418eb1921dd605fbf2b0
0115 14:52:11.982608 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.119342 (+136734us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.204879 (+ 85537us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.268498 (+ 63619us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1298006}
I20250115 14:52:12.268764 2563 raft_consensus.cc:513] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.269224 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 5b85d63974ab418eb1921dd605fbf2b0 (table default.Table_854a113be33b4b698a50ae5bd10a2fff [id=f1ce643dc59040148201fdd0c049fb20]) successfully deleted
I20250115 14:52:12.269353 2563 leader_election.cc:290] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.269497 1167 tablet_replica.cc:331] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:12.269665 1167 raft_consensus.cc:2238] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.269786 1167 raft_consensus.cc:2267] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.269729 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b599c6b8174959b8b66e90e9242057" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:12.269883 913 raft_consensus.cc:3054] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.270284 1167 ts_tablet_manager.cc:1905] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.270741 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27a88650b024af19938618d9ed0cc23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.271224 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc659e2d2d943a99aef530e7d71df88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.271589 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e33ebec8b64f2398a7f1003b15cee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.271759 1046 raft_consensus.cc:2463] T dc363f70f3f44d309f69fac9356c6257 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:12.272213 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da4a558e1ea4bccb868ad5c489e96d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.272405 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23318b1ad0774a76880def17a64ac059. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.272647 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0567f95a3444fedb002e550615a99fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.272789 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0163895f3b274712ba69890f7e49ab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.273123 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed0b0ccdcff423d9bb2b9a83511616d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.273481 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5018e358d234ea28023da76a8c61a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.273813 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addb4972a8de427d99b0cbb646796c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.274161 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3c12a0a7eb42c5888961103f7eb178. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.274581 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93aaea90b0924327807e7f594154b14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.274600 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f4d5908220b548f1ba60227f435a948a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.275123 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4d5908220b548f1ba60227f435a948a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.275103 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b599c6b8174959b8b66e90e9242057" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:12.275733 984 leader_election.cc:343] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.272241 913 raft_consensus.cc:2463] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:12.276327 983 leader_election.cc:304] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.276726 2563 raft_consensus.cc:2798] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:12.276829 2563 raft_consensus.cc:695] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:12.276898 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9ab0ec2875a044198bb7906f36b7680b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.277067 2563 consensus_queue.cc:237] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:12.277359 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ab0ec2875a044198bb7906f36b7680b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.278388 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4607c9a022124589b931dfbb075cefb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.278757 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4607c9a022124589b931dfbb075cefb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.281412 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9e7c8e59aa4ea59bad35c8a7591a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.281641 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet e49dc97eea474252a5e0ef4af6e015a9 (table default.Table_4f00ef2ef63d4f77929a245aa9bff035 [id=8502b5fe264f4fb5a89bd4a91949c90b]). Expected version 1 got 0
I20250115 14:52:12.281915 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a300a76b8f7942dea648a20cbb3d5368 (table default.Table_0034e4c667b44bbb83a5a93e321e5113 [id=f6bae7686ebc45488f5eb27a554d28a3]). Expected version 2 got 0
I20250115 14:52:12.285192 847 leader_election.cc:304] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.285477 2341 raft_consensus.cc:2798] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:12.285593 2341 raft_consensus.cc:695] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:12.285846 2341 consensus_queue.cc:237] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.288781 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed0b0ccdcff423d9bb2b9a83511616d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.289270 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc659e2d2d943a99aef530e7d71df88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.291414 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0567f95a3444fedb002e550615a99fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.291841 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addb4972a8de427d99b0cbb646796c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.292246 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0163895f3b274712ba69890f7e49ab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.292659 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5018e358d234ea28023da76a8c61a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.293040 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9e7c8e59aa4ea59bad35c8a7591a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.293419 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27a88650b024af19938618d9ed0cc23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.295670 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1021) took 1640 ms. Trace:
I20250115 14:52:12.295754 1751 rpcz_store.cc:276] 0115 14:52:10.655060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.655568 (+ 508us) service_pool.cc:225] Handling call
0115 14:52:12.155667 (+1500099us) ts_tablet_manager.cc:1145] Deleting tablet 5265d5eeac83481ca97801e3475886da
0115 14:52:12.155675 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.191244 (+ 35569us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.221687 (+ 30443us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.295651 (+ 73964us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499995}
I20250115 14:52:12.296066 1751 tablet_replica.cc:331] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:12.296260 1751 raft_consensus.cc:2238] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.296406 1751 raft_consensus.cc:2267] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.296904 1751 ts_tablet_manager.cc:1905] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.316579 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 5265d5eeac83481ca97801e3475886da (table default.Table_4add05103df84280812bf61910d76d1d [id=47db9a9013ec4a2cb3cc0cc8f9caba12]) successfully deleted
I20250115 14:52:12.345983 2403 raft_consensus.cc:491] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.346166 2403 raft_consensus.cc:513] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.346788 2403 leader_election.cc:290] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:12.347918 710 leader_election.cc:343] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 95718d39a7824517a36c49c3b86fef30
W20250115 14:52:12.350121 710 leader_election.cc:343] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 95718d39a7824517a36c49c3b86fef30
I20250115 14:52:12.350245 710 leader_election.cc:304] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:12.350518 2403 raft_consensus.cc:2743] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.368199 1166 ts_tablet_manager.cc:1918] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:12.368350 1166 log.cc:1198] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/377d711f052545ddbe499afe7ba3de28
I20250115 14:52:12.368661 1166 ts_tablet_manager.cc:1939] T 377d711f052545ddbe499afe7ba3de28 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:12.376425 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d27a88650b024af19938618d9ed0cc23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.376652 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93aaea90b0924327807e7f594154b14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.376806 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5018e358d234ea28023da76a8c61a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.376960 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7da4a558e1ea4bccb868ad5c489e96d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.377115 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed0b0ccdcff423d9bb2b9a83511616d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.377291 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec9e7c8e59aa4ea59bad35c8a7591a79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.377489 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0163895f3b274712ba69890f7e49ab9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.377668 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0567f95a3444fedb002e550615a99fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.377874 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addb4972a8de427d99b0cbb646796c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.378365 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e33ebec8b64f2398a7f1003b15cee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.378739 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3c12a0a7eb42c5888961103f7eb178. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.379297 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc659e2d2d943a99aef530e7d71df88. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.380882 891 tablet_service.cc:1467] Processing CreateTablet for tablet 11f1102485c344308b12848bca4a6617 (DEFAULT_TABLE table=default.Table_9cae3ab41ae3426081cc7cf3a3a05704 [id=2327eb0bb58044469d926eff95cd1c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.380919 1020 tablet_service.cc:1467] Processing CreateTablet for tablet c8e33ebec8b64f2398a7f1003b15cee7 (DEFAULT_TABLE table=default.Table_89373c553f9c407a9e205bfe0d1acf99 [id=26c53728ab22430b95ca322543118a43]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.381340 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e33ebec8b64f2398a7f1003b15cee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.385223 751 tablet_service.cc:1467] Processing CreateTablet for tablet 11f1102485c344308b12848bca4a6617 (DEFAULT_TABLE table=default.Table_9cae3ab41ae3426081cc7cf3a3a05704 [id=2327eb0bb58044469d926eff95cd1c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.387885 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 11f1102485c344308b12848bca4a6617 (DEFAULT_TABLE table=default.Table_9cae3ab41ae3426081cc7cf3a3a05704 [id=2327eb0bb58044469d926eff95cd1c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.391126 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d02b19d9fa214584a8c9a02041a6da29" } new_table_name: "default.Table_bbeda353ef0a4bc987d588b99ee57db2" modify_external_catalogs: true
I20250115 14:52:12.393122 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 4adf1353bd554d8294a7f570931095d2 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:12.393568 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9105e59402544b38a833063271631821" } new_table_name: "default.Table_f4536e6b631a4f40acf32c7fe4c42f10" modify_external_catalogs: true
I20250115 14:52:12.393851 885 tablet_service.cc:1514] Processing DeleteTablet for tablet b803d8e103a5423f85c75c9006809b17 with delete_type TABLET_DATA_DELETED (replaced by tablet 6437873733584722979c1e9ae8b0730b) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.394109 890 tablet_service.cc:1467] Processing CreateTablet for tablet ea91cd062df445b1aed8007dd779f461 (DEFAULT_TABLE table=default.Table_8e7481bab16b4d808c56d4a6b10c11e5 [id=8a40356fe90244c782eb416b67faa611]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.396473 893 tablet_service.cc:1467] Processing CreateTablet for tablet 50f8d93d655a447cb9e92997342a61a7 (DEFAULT_TABLE table=default.Table_01627515f7f64b1eb74dcf308bc61ea6 [id=15bd81f7d102442a83901b70344d3755]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.396796 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 580742eb5dac4dba9995809890e0e315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.395290 737 tablet_service.cc:1467] Processing CreateTablet for tablet ac97cd8e11cf4b5e8978902d1c99aad8 (DEFAULT_TABLE table=default.Table_19880e28f59446b0a4b4a8cc082428b7 [id=28e483eaf07c438d97a979706cb3195b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.395319 753 tablet_service.cc:1467] Processing CreateTablet for tablet 664d5fbc6f3c43119667919acde07bcd (DEFAULT_TABLE table=default.Table_92ea5828cb8a4d22b8965d819411004d [id=f9a2e7e26b144ebab161e56a61652fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.394800 745 tablet_service.cc:1467] Processing CreateTablet for tablet 50f8d93d655a447cb9e92997342a61a7 (DEFAULT_TABLE table=default.Table_01627515f7f64b1eb74dcf308bc61ea6 [id=15bd81f7d102442a83901b70344d3755]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.397522 747 tablet_service.cc:1467] Processing CreateTablet for tablet ea91cd062df445b1aed8007dd779f461 (DEFAULT_TABLE table=default.Table_8e7481bab16b4d808c56d4a6b10c11e5 [id=8a40356fe90244c782eb416b67faa611]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.404359 1026 tablet_service.cc:1467] Processing CreateTablet for tablet ea91cd062df445b1aed8007dd779f461 (DEFAULT_TABLE table=default.Table_8e7481bab16b4d808c56d4a6b10c11e5 [id=8a40356fe90244c782eb416b67faa611]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.405164 886 tablet_service.cc:1467] Processing CreateTablet for tablet 664d5fbc6f3c43119667919acde07bcd (DEFAULT_TABLE table=default.Table_92ea5828cb8a4d22b8965d819411004d [id=f9a2e7e26b144ebab161e56a61652fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.405761 1017 tablet_service.cc:1467] Processing CreateTablet for tablet ac97cd8e11cf4b5e8978902d1c99aad8 (DEFAULT_TABLE table=default.Table_19880e28f59446b0a4b4a8cc082428b7 [id=28e483eaf07c438d97a979706cb3195b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.406088 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 50f8d93d655a447cb9e92997342a61a7 (DEFAULT_TABLE table=default.Table_01627515f7f64b1eb74dcf308bc61ea6 [id=15bd81f7d102442a83901b70344d3755]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.406185 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 664d5fbc6f3c43119667919acde07bcd (DEFAULT_TABLE table=default.Table_92ea5828cb8a4d22b8965d819411004d [id=f9a2e7e26b144ebab161e56a61652fbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.406853 876 tablet_service.cc:1467] Processing CreateTablet for tablet ac97cd8e11cf4b5e8978902d1c99aad8 (DEFAULT_TABLE table=default.Table_19880e28f59446b0a4b4a8cc082428b7 [id=28e483eaf07c438d97a979706cb3195b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.407630 2563 raft_consensus.cc:491] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.407792 2563 raft_consensus.cc:513] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.408795 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a300a76b8f7942dea648a20cbb3d5368" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:12.409348 984 leader_election.cc:343] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.409986 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a300a76b8f7942dea648a20cbb3d5368" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:12.410611 983 leader_election.cc:343] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.410704 983 leader_election.cc:304] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.410943 2564 raft_consensus.cc:2743] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.411181 2563 leader_election.cc:290] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.455638 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea91cd062df445b1aed8007dd779f461. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:12.456087 2296 log.cc:926] Time spent T 00000000000000000000000000000000 P 7d97cf5f1fbb4bc6883711020e2fa3f3: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20250115 14:52:12.457463 2611 log.cc:926] Time spent T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20250115 14:52:12.458864 1168 tablet_bootstrap.cc:492] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.458993 1168 ts_tablet_manager.cc:1397] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.299s user 0.010s sys 0.002s
I20250115 14:52:12.459717 1168 raft_consensus.cc:357] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.459901 1168 raft_consensus.cc:383] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.459947 1168 raft_consensus.cc:738] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.460131 1168 consensus_queue.cc:260] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.460603 1168 ts_tablet_manager.cc:1428] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:12.460855 1168 tablet_bootstrap.cc:492] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.462416 1168 tablet_bootstrap.cc:654] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:12.463575 2612 log.cc:926] Time spent T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250115 14:52:12.469355 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 664d5fbc6f3c43119667919acde07bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.469821 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac97cd8e11cf4b5e8978902d1c99aad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.469990 1168 tablet_bootstrap.cc:492] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.470108 1168 ts_tablet_manager.cc:1397] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20250115 14:52:12.470777 1168 raft_consensus.cc:357] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.470952 1168 raft_consensus.cc:383] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.470994 1168 raft_consensus.cc:738] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.471176 1168 consensus_queue.cc:260] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.472523 1164 tablet_bootstrap.cc:654] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.473310 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f8d93d655a447cb9e92997342a61a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.473738 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea91cd062df445b1aed8007dd779f461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.476328 1166 tablet_replica.cc:331] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:12.476404 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f1102485c344308b12848bca4a6617. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.476501 1166 raft_consensus.cc:2238] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.476612 1166 raft_consensus.cc:2267] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.476768 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac97cd8e11cf4b5e8978902d1c99aad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.477047 1166 ts_tablet_manager.cc:1905] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.477141 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 664d5fbc6f3c43119667919acde07bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.490914 1167 ts_tablet_manager.cc:1918] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.491063 1167 log.cc:1198] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/742792be08444792892bd9cd7336dda3
I20250115 14:52:12.491351 1167 ts_tablet_manager.cc:1939] T 742792be08444792892bd9cd7336dda3 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:12.492084 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea91cd062df445b1aed8007dd779f461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.492281 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f8d93d655a447cb9e92997342a61a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.492514 2549 raft_consensus.cc:491] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.492578 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac97cd8e11cf4b5e8978902d1c99aad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.492622 2549 raft_consensus.cc:513] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.492895 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 664d5fbc6f3c43119667919acde07bcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.493223 2549 leader_election.cc:290] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.493331 2549 raft_consensus.cc:491] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.493414 2549 raft_consensus.cc:513] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.493606 1166 ts_tablet_manager.cc:1918] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.493691 1166 log.cc:1198] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/6942d102069d4c33b23fa2c0b29f94a6
I20250115 14:52:12.493934 1166 ts_tablet_manager.cc:1939] T 6942d102069d4c33b23fa2c0b29f94a6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:12.493919 2549 leader_election.cc:290] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.494100 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f1102485c344308b12848bca4a6617. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.494692 1166 tablet_replica.cc:331] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:12.494849 1166 raft_consensus.cc:2238] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.494956 1166 raft_consensus.cc:2267] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.495395 1166 ts_tablet_manager.cc:1905] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.495918 2564 raft_consensus.cc:491] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.496064 2564 raft_consensus.cc:513] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.496706 2564 leader_election.cc:290] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.497350 1168 ts_tablet_manager.cc:1428] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250115 14:52:12.497582 1168 tablet_bootstrap.cc:492] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.499408 1168 tablet_bootstrap.cc:654] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.500444 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50f8d93d655a447cb9e92997342a61a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.500901 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f1102485c344308b12848bca4a6617. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.502033 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4c2707915e4e97ad6c66cb249c6d9e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.505463 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4c2707915e4e97ad6c66cb249c6d9e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:12.505936 983 leader_election.cc:343] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.507516 2564 raft_consensus.cc:491] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.507629 2564 raft_consensus.cc:513] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.508220 2564 leader_election.cc:290] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.508744 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3f8cae6244968927b864e7ce97565" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:12.508908 913 raft_consensus.cc:2463] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:12.510859 1166 ts_tablet_manager.cc:1918] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.510864 983 leader_election.cc:304] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:12.510965 1166 log.cc:1198] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/95718d39a7824517a36c49c3b86fef30
I20250115 14:52:12.511112 2564 raft_consensus.cc:2798] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:12.511178 2564 raft_consensus.cc:491] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:12.511229 1166 ts_tablet_manager.cc:1939] T 95718d39a7824517a36c49c3b86fef30 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:12.511230 2564 raft_consensus.cc:3054] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.512058 1166 tablet_replica.cc:331] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:12.512231 1166 raft_consensus.cc:2238] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.512373 1166 raft_consensus.cc:2267] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.512850 1166 ts_tablet_manager.cc:1905] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.513908 2564 raft_consensus.cc:513] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.514447 2564 leader_election.cc:290] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.515007 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3f8cae6244968927b864e7ce97565" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:12.515161 913 raft_consensus.cc:3054] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.517038 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3f8cae6244968927b864e7ce97565" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:12.517266 1048 raft_consensus.cc:2388] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 73c86158cc2641f182ea50a06634592d in current term 1: Already voted for candidate c238c037a1c74725a001e2b4a1ff4304 in this term.
I20250115 14:52:12.517594 913 raft_consensus.cc:2463] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:12.525647 1166 ts_tablet_manager.cc:1918] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.529721 1166 log.cc:1198] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/83415f1c07ad495cabd287bdcd54dad4
I20250115 14:52:12.530021 1166 ts_tablet_manager.cc:1939] T 83415f1c07ad495cabd287bdcd54dad4 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:12.538187 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0651be54e0b244a4bace9f47efc9b58a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.539376 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3f8cae6244968927b864e7ce97565" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:12.539417 847 leader_election.cc:343] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 0651be54e0b244a4bace9f47efc9b58a
I20250115 14:52:12.540130 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3f8cae6244968927b864e7ce97565" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:12.542433 847 consensus_peers.cc:487] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2a11b7b0dd6494daf7b017edd4317a5. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:12.542879 848 leader_election.cc:343] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.542933 1164 tablet_bootstrap.cc:492] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:12.542977 848 leader_election.cc:304] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.543041 1164 ts_tablet_manager.cc:1397] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.275s user 0.002s sys 0.000s
I20250115 14:52:12.543169 2341 raft_consensus.cc:2743] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:12.543331 848 leader_election.cc:343] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.543402 848 leader_election.cc:304] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.543548 2341 raft_consensus.cc:2743] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.543699 1164 raft_consensus.cc:357] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.543875 1164 raft_consensus.cc:383] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.543917 1164 raft_consensus.cc:738] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.544080 1164 consensus_queue.cc:260] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.544162 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 377d711f052545ddbe499afe7ba3de28 (table default.Table_7ec0e7f2e4de4b14bd6cea8489658118 [id=898cecdad4254997a5bed59d46f7acec]) successfully deleted
I20250115 14:52:12.544524 1164 ts_tablet_manager.cc:1428] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20250115 14:52:12.544534 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a696017e859f474bb1fd062df44c224d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a696017e859f474bb1fd062df44c224d already in progress: creating tablet
W20250115 14:52:12.544673 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 36ac2c9c80824c25bb974092b3aba744 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36ac2c9c80824c25bb974092b3aba744 already in progress: creating tablet
I20250115 14:52:12.544756 1164 tablet_bootstrap.cc:492] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:12.544780 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c23b26e4b7ff4c699d1c86adf6731c71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c23b26e4b7ff4c699d1c86adf6731c71 already in progress: creating tablet
W20250115 14:52:12.544886 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c6cb8dfa7c42480bbd139d5acf059a57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6cb8dfa7c42480bbd139d5acf059a57 already in progress: creating tablet
W20250115 14:52:12.544989 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ad417f06081d41248dff6f1eed175f29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad417f06081d41248dff6f1eed175f29 already in progress: creating tablet
W20250115 14:52:12.545105 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3be094dcd3cb4f9d9259d354300c1600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3be094dcd3cb4f9d9259d354300c1600 already in progress: creating tablet
I20250115 14:52:12.545531 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 6942d102069d4c33b23fa2c0b29f94a6 (table default.Table_25a5a732ec814f24893d09a6f0b38d2c [id=fa886a7e8fcd43a096892c6435e13c29]) successfully deleted
I20250115 14:52:12.546164 1164 tablet_bootstrap.cc:654] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.549194 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6f3f8cae6244968927b864e7ce97565" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.549904 2610 tablet.cc:1721] T 92aeca346a7746468dbbff053d933472 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250115 14:52:12.551246 984 leader_election.cc:343] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.551367 984 leader_election.cc:304] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:12.551760 984 leader_election.cc:343] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.551937 2273 raft_consensus.cc:2743] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.552994 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4e39e13fb8c84ae18bac5e45d576a0ab
I20250115 14:52:12.553280 983 leader_election.cc:304] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.553905 2469 raft_consensus.cc:2798] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:12.554037 2469 raft_consensus.cc:695] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:12.554106 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 546fe492182f47b7b008f4719f97e877
I20250115 14:52:12.554243 2469 consensus_queue.cc:237] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.555233 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9022738b53c74ce99e8f35adb655d200
I20250115 14:52:12.555907 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 86c04fa23e7c40a5b5e07852e5d74a27
I20250115 14:52:12.558532 1164 tablet_bootstrap.cc:492] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:12.558648 1164 ts_tablet_manager.cc:1397] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250115 14:52:12.559319 1164 raft_consensus.cc:357] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.559513 1164 raft_consensus.cc:383] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.559589 1164 raft_consensus.cc:738] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.559805 1164 consensus_queue.cc:260] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.560261 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e39e13fb8c84ae18bac5e45d576a0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.560329 1164 ts_tablet_manager.cc:1428] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:12.560475 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 546fe492182f47b7b008f4719f97e877 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.560645 1164 tablet_bootstrap.cc:492] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:12.560645 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 9022738b53c74ce99e8f35adb655d200 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.560787 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 86c04fa23e7c40a5b5e07852e5d74a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:12.562028 984 leader_election.cc:343] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.562196 1162 tablet_bootstrap.cc:492] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.562297 1162 ts_tablet_manager.cc:1397] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.305s user 0.003s sys 0.001s
I20250115 14:52:12.562891 1162 raft_consensus.cc:357] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.563057 1162 raft_consensus.cc:383] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.563099 1162 raft_consensus.cc:738] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.563251 1162 consensus_queue.cc:260] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.563680 1162 ts_tablet_manager.cc:1428] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250115 14:52:12.563941 1162 tablet_bootstrap.cc:492] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.564491 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 9022738b53c74ce99e8f35adb655d200 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.564790 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 86c04fa23e7c40a5b5e07852e5d74a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.565459 1162 tablet_bootstrap.cc:654] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.567600 1164 tablet_bootstrap.cc:654] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.568877 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 546fe492182f47b7b008f4719f97e877 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.569515 2403 raft_consensus.cc:491] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.569635 2403 raft_consensus.cc:513] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.570189 2403 leader_election.cc:290] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.570773 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1ce59e05054a009cd36e12db35d449" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:12.570955 909 raft_consensus.cc:2463] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 0.
I20250115 14:52:12.571341 710 leader_election.cc:304] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters:
I20250115 14:52:12.571600 2403 raft_consensus.cc:2798] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:12.571671 2403 raft_consensus.cc:491] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:12.571729 2403 raft_consensus.cc:3054] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.574056 2403 raft_consensus.cc:513] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:12.575042 710 leader_election.cc:343] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 4f1ce59e05054a009cd36e12db35d449
I20250115 14:52:12.575387 2403 leader_election.cc:290] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Requested vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:12.575443 710 leader_election.cc:343] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 4f1ce59e05054a009cd36e12db35d449
I20250115 14:52:12.577531 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1ce59e05054a009cd36e12db35d449" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:12.577697 909 raft_consensus.cc:3054] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.580016 909 raft_consensus.cc:2463] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d7ecece2ab49e1be7079216517d13b in term 1.
I20250115 14:52:12.580760 710 leader_election.cc:304] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b; no voters: c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:12.580855 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d5bce1994c5a4329bb30179b053babbb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.581249 2403 raft_consensus.cc:2798] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:52:12.581437 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5bce1994c5a4329bb30179b053babbb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.582069 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_35a3371b3a6c4f35b2990f828671681e [id=60ff7f730d394990a0263e322d28d7ff] alter complete (version 4)
I20250115 14:52:12.582873 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_52e9521f04f342acb2bad7d736f8847e" } new_table_name: "default.Table_cc9a98342a67450a92d9f329a3cfc05e" modify_external_catalogs: true
I20250115 14:52:12.583463 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_49e2a633e2274a808b994184c4d5d28b" } new_table_name: "default.Table_fd84fd3d47974e05a580fae352838c20" modify_external_catalogs: true
I20250115 14:52:12.584424 2549 raft_consensus.cc:491] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.584537 2549 raft_consensus.cc:513] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.585088 2549 leader_election.cc:290] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.586051 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1021) took 1782 ms. Trace:
I20250115 14:52:12.586155 1167 rpcz_store.cc:276] 0115 14:52:10.803796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.803851 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:12.269453 (+1465602us) ts_tablet_manager.cc:1145] Deleting tablet 742792be08444792892bd9cd7336dda3
0115 14:52:12.269462 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.476272 (+206810us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.490845 (+ 14573us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.586004 (+ 95159us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465399}
I20250115 14:52:12.586464 1167 tablet_replica.cc:331] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:12.586648 1167 raft_consensus.cc:2238] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:12.586925 1167 pending_rounds.cc:62] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:12.586982 1167 pending_rounds.cc:69] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558388334370816 op_type: NO_OP noop_request { }
I20250115 14:52:12.587080 1167 raft_consensus.cc:2883] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:12.587158 1167 raft_consensus.cc:2267] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.587664 1167 ts_tablet_manager.cc:1905] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.588572 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7e70288eb248f1a71646c66e376a63" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:12.588650 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 92aeca346a7746468dbbff053d933472 (table default.Table_35a3371b3a6c4f35b2990f828671681e [id=60ff7f730d394990a0263e322d28d7ff]). Expected version 4 got 0
I20250115 14:52:12.588773 1046 raft_consensus.cc:2463] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:12.589452 847 leader_election.cc:304] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:12.589910 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 2560d42c15c64b4c9c0cfe95706b4d9e from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:12.590018 2341 raft_consensus.cc:2798] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:12.590104 2341 raft_consensus.cc:491] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:12.590178 2341 raft_consensus.cc:3054] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.590417 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7c4783895e7c4f9da20efe43779ce1e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.591112 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0ac18ecc16b2424881834e142ba358d9" } new_table_name: "default.Table_a612d89e3cfb4e96a84098477bdd6f84" modify_external_catalogs: true
I20250115 14:52:12.592268 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3089c6f63edb4642b0ed0a82ee1b5051"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.592813 2341 raft_consensus.cc:513] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.593374 2341 leader_election.cc:290] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.593305 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bc62cb62b8884088aa71a60746b07ffd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.593683 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc62cb62b8884088aa71a60746b07ffd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:12.594691 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3089c6f63edb4642b0ed0a82ee1b5051 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:12.596742 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:45653, user_credentials={real_user=slave}} blocked reactor thread for 50912.4us
I20250115 14:52:12.596753 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7e70288eb248f1a71646c66e376a63" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.599228 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 95718d39a7824517a36c49c3b86fef30 (table default.Table_5d8548b30284461086fdcd351454ad57 [id=d114031263ea4c87a6c4d12c9d6a0d32]) successfully deleted
I20250115 14:52:12.599376 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 83415f1c07ad495cabd287bdcd54dad4 (table default.Table_458a573b4a4b4714950723b426f0d265 [id=1fd54f7dafdc4777bd06982391452889]) successfully deleted
W20250115 14:52:12.599514 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 18f23815196047628d562da7312b010f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18f23815196047628d562da7312b010f already in progress: creating tablet
W20250115 14:52:12.599645 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0fe88a1365954ef5beb1d67b1cc4b747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe88a1365954ef5beb1d67b1cc4b747 already in progress: creating tablet
I20250115 14:52:12.600100 1167 ts_tablet_manager.cc:1918] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:12.600212 1167 log.cc:1198] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b2a11b7b0dd6494daf7b017edd4317a5
I20250115 14:52:12.600790 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 86c04fa23e7c40a5b5e07852e5d74a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.600906 744 tablet_service.cc:1514] Processing DeleteTablet for tablet 9022738b53c74ce99e8f35adb655d200 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.600981 737 tablet_service.cc:1514] Processing DeleteTablet for tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 with delete_type TABLET_DATA_DELETED (replaced by tablet bc1440e3cd444cf8b3365a0af8de8958) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.601086 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 546fe492182f47b7b008f4719f97e877 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.601110 753 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9b57b8b9434f55ace5a4f47380df54 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f8bae9867d745afb1dfdf47bcb881da) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.601222 2564 raft_consensus.cc:491] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.601294 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7e70288eb248f1a71646c66e376a63" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:12.601334 1751 ts_tablet_manager.cc:1918] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.601408 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e39e13fb8c84ae18bac5e45d576a0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.601439 1046 raft_consensus.cc:3054] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:12.601337 2564 raft_consensus.cc:513] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.602605 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4475441e4a467a8d593b742d2a5b80" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:12.603085 984 leader_election.cc:343] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.603364 2564 leader_election.cc:290] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.603941 1046 raft_consensus.cc:2463] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:12.603955 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4475441e4a467a8d593b742d2a5b80" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:12.604370 847 leader_election.cc:304] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:52:12.604408 983 leader_election.cc:343] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.604486 983 leader_election.cc:304] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.604609 2549 raft_consensus.cc:2798] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:12.604727 2549 raft_consensus.cc:695] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:12.604800 2564 raft_consensus.cc:2743] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.601457 1751 log.cc:1198] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/1140016d11a74a2cadba494c0b319c54
I20250115 14:52:12.604993 2549 consensus_queue.cc:237] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.605401 1751 ts_tablet_manager.cc:1939] T 1140016d11a74a2cadba494c0b319c54 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:12.607270 848 leader_election.cc:343] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.607692 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 742792be08444792892bd9cd7336dda3 (table default.Table_97c066067f7a49cab2dd5bdade9f6de2 [id=8c1c9324a0cb4ce5b5f64af989e4bdf6]) successfully deleted
I20250115 14:52:12.610374 2549 raft_consensus.cc:491] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.610498 2549 raft_consensus.cc:513] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.610875 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7e70288eb248f1a71646c66e376a63" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:12.611024 2549 leader_election.cc:290] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:12.611521 848 leader_election.cc:343] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.611660 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a6f83730f74d13823bd15d93538df7" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.611986 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8bae9867d745afb1dfdf47bcb881da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.612165 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet ded663ac1197454bbde3c4c2d526002d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.612318 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet a48a0af9422f43cf86531665c0fa1aee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.612455 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet e244689cef6844e2935becd265eecad0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1a69b6de5d433f82579b1259548fd2) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.613976 1166 tablet_replica.cc:331] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:12.614195 1166 raft_consensus.cc:2238] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.614331 1166 raft_consensus.cc:2267] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.614603 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1022) took 1959 ms. Trace:
I20250115 14:52:12.614687 1751 rpcz_store.cc:276] 0115 14:52:10.655140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.655694 (+ 554us) service_pool.cc:225] Handling call
0115 14:52:12.296008 (+1640314us) ts_tablet_manager.cc:1145] Deleting tablet 1140016d11a74a2cadba494c0b319c54
0115 14:52:12.296017 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.517576 (+221559us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.575170 (+ 57594us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.614585 (+ 39415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1640226}
I20250115 14:52:12.614810 1166 ts_tablet_manager.cc:1905] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.615053 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1023) took 1959 ms. Trace:
I20250115 14:52:12.615123 1751 rpcz_store.cc:276] 0115 14:52:10.655212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.655808 (+ 596us) service_pool.cc:225] Handling call
0115 14:52:12.614948 (+1959140us) ts_tablet_manager.cc:1145] Deleting tablet 13053264b5ca41dd85e53d928aa511fd
0115 14:52:12.614957 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.615038 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1959039}
I20250115 14:52:12.615170 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 1140016d11a74a2cadba494c0b319c54 (table default.Table_06fdf72400894754b2da4a05ae517fac [id=fa2c05b8018545958699fafc77f6e832]) successfully deleted
I20250115 14:52:12.615355 1751 tablet_replica.cc:331] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:12.615505 1751 raft_consensus.cc:2238] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.615645 1751 raft_consensus.cc:2267] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:12.615634 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 13053264b5ca41dd85e53d928aa511fd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13053264b5ca41dd85e53d928aa511fd already in progress: creating tablet
I20250115 14:52:12.616094 1751 ts_tablet_manager.cc:1905] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.617019 1167 ts_tablet_manager.cc:1939] T b2a11b7b0dd6494daf7b017edd4317a5 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:12.617789 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1029) took 1750 ms. Trace:
I20250115 14:52:12.617926 1167 rpcz_store.cc:276] 0115 14:52:10.867416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.867474 (+ 58us) service_pool.cc:225] Handling call
0115 14:52:12.586400 (+1718926us) ts_tablet_manager.cc:1145] Deleting tablet b2a11b7b0dd6494daf7b017edd4317a5
0115 14:52:12.586409 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.597228 (+ 10819us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.599991 (+ 2763us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.617730 (+ 17739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1718764}
I20250115 14:52:12.618366 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1028) took 1765 ms. Trace:
I20250115 14:52:12.618953 1167 rpcz_store.cc:276] 0115 14:52:10.852755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.877789 (+ 25034us) service_pool.cc:225] Handling call
0115 14:52:12.618246 (+1740457us) ts_tablet_manager.cc:1145] Deleting tablet 5f0eeede8f714b92b1e3a9a558289c3e
0115 14:52:12.618255 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.618335 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1740280}
I20250115 14:52:12.619724 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1032) took 1720 ms. Trace:
W20250115 14:52:12.619927 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c4783895e7c4f9da20efe43779ce1e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.623310 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b2a11b7b0dd6494daf7b017edd4317a5 (table default.Table_8e068b869e304185abaccf2d63685cd3 [id=5a50430da9af480282aa31475f196737]) successfully deleted
W20250115 14:52:12.623524 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 5f0eeede8f714b92b1e3a9a558289c3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f0eeede8f714b92b1e3a9a558289c3e already in progress: creating tablet
W20250115 14:52:12.620388 847 leader_election.cc:343] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 65a6f83730f74d13823bd15d93538df7
I20250115 14:52:12.621546 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e39e13fb8c84ae18bac5e45d576a0ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:12.625169 848 leader_election.cc:343] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.625466 848 leader_election.cc:304] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.625909 2549 raft_consensus.cc:2743] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.625952 1162 tablet_bootstrap.cc:492] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.626032 1166 ts_tablet_manager.cc:1918] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.626134 1166 log.cc:1198] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/c37f4c2ab9d248a1af817cfa2c2e2ff0
I20250115 14:52:12.626252 1167 rpcz_store.cc:276] 0115 14:52:10.898769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.899134 (+ 365us) service_pool.cc:225] Handling call
0115 14:52:12.619595 (+1720461us) ts_tablet_manager.cc:1145] Deleting tablet 8d4c30bccb3549e6963dfbbecf68cbbe
0115 14:52:12.619604 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.619681 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1720243}
I20250115 14:52:12.626443 1166 ts_tablet_manager.cc:1939] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:52:12.626659 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 86c04fa23e7c40a5b5e07852e5d74a27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86c04fa23e7c40a5b5e07852e5d74a27 already in progress: creating tablet
W20250115 14:52:12.626906 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9022738b53c74ce99e8f35adb655d200 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9022738b53c74ce99e8f35adb655d200 already in progress: creating tablet
I20250115 14:52:12.627465 1166 tablet_replica.cc:331] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:12.627645 1166 raft_consensus.cc:2238] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.627770 1166 raft_consensus.cc:2267] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.628381 1166 ts_tablet_manager.cc:1905] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.629686 1162 ts_tablet_manager.cc:1397] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.066s user 0.001s sys 0.005s
I20250115 14:52:12.632628 1162 raft_consensus.cc:357] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.632807 1162 raft_consensus.cc:383] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.632856 1162 raft_consensus.cc:738] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.633333 1162 consensus_queue.cc:260] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:12.633482 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2087b4ce466f4167b7c32c7074dcf6e7 consensus state has no leader
I20250115 14:52:12.633795 1162 ts_tablet_manager.cc:1428] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
W20250115 14:52:12.634101 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d27a88650b024af19938618d9ed0cc23 consensus state has no leader
I20250115 14:52:12.634315 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 856533ed254a49b4a461ed1ad3eb405e from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:12.634763 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 7233ebf3bae1462d8eeb3669b39944c0 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:12.635432 2469 raft_consensus.cc:491] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.635535 2469 raft_consensus.cc:513] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:12.635927 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a739d36ef9d54c089667ff32de875af6 consensus state has no leader
I20250115 14:52:12.636052 2469 leader_election.cc:290] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.636402 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1035) took 1654 ms. Trace:
I20250115 14:52:12.636502 1167 rpcz_store.cc:276] 0115 14:52:10.981975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.982350 (+ 375us) service_pool.cc:225] Handling call
0115 14:52:12.636220 (+1653870us) ts_tablet_manager.cc:1145] Deleting tablet 42d8c79e0f7a444c869378678344f2b7
0115 14:52:12.636232 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.636355 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1653685}
I20250115 14:52:12.636551 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398ac8af619c427287b3964443173864" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.636667 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398ac8af619c427287b3964443173864" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:12.636973 984 leader_election.cc:343] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:12.637153 983 leader_election.cc:343] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.637254 983 leader_election.cc:304] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.637591 2469 raft_consensus.cc:2743] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.638105 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1034) took 1658 ms. Trace:
I20250115 14:52:12.638192 1167 rpcz_store.cc:276] 0115 14:52:10.979869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.983112 (+ 3243us) service_pool.cc:225] Handling call
0115 14:52:12.637980 (+1654868us) ts_tablet_manager.cc:1145] Deleting tablet 638507d08f654011bea2597c22a788a2
0115 14:52:12.637989 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.638071 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654677}
I20250115 14:52:12.642976 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1036) took 1632 ms. Trace:
I20250115 14:52:12.643091 1167 rpcz_store.cc:276] 0115 14:52:11.010788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.014229 (+ 3441us) service_pool.cc:225] Handling call
0115 14:52:12.642839 (+1628610us) ts_tablet_manager.cc:1145] Deleting tablet 36de948083a946eba1a96f276e4f4f91
0115 14:52:12.642848 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.642930 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1628409}
I20250115 14:52:12.638945 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 (table default.Table_6f0b6c724c1c43e7adf538ad7369b7a1 [id=9e8ce51e784e4bdbb3ac78cba10bfedf]) successfully deleted
I20250115 14:52:12.636282 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ee7134670e574f279ad4f5557f395e69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.647125 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 546fe492182f47b7b008f4719f97e877 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 546fe492182f47b7b008f4719f97e877 already in progress: creating tablet
W20250115 14:52:12.647302 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee7134670e574f279ad4f5557f395e69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:12.644562 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ace1012dde2b4ecbb9e58e1ad608644f consensus state has no leader
I20250115 14:52:12.648577 2403 raft_consensus.cc:695] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Becoming Leader. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Running, Role: LEADER
I20250115 14:52:12.648752 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ca37f38ca7de45edba3a3cbcfae56bd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.649009 2403 consensus_queue.cc:237] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:12.649183 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca37f38ca7de45edba3a3cbcfae56bd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.651037 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1037) took 1362 ms. Trace:
I20250115 14:52:12.651497 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_262d362a9eb447c689cbb66e877ab7be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.651806 1167 rpcz_store.cc:276] 0115 14:52:11.289020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.289069 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:12.650896 (+1361827us) ts_tablet_manager.cc:1145] Deleting tablet 288b0e455a854464940f65a2ab909d0b
0115 14:52:12.650907 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.650992 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1361668}
W20250115 14:52:12.651916 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_262d362a9eb447c689cbb66e877ab7be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:12.637967 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 8d4c30bccb3549e6963dfbbecf68cbbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d4c30bccb3549e6963dfbbecf68cbbe already in progress: creating tablet
I20250115 14:52:12.645020 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_eb0e928b8265482990bb5fbf1072985f" } modify_external_catalogs: true
I20250115 14:52:12.653247 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_eb0e928b8265482990bb5fbf1072985f" } modify_external_catalogs: true
W20250115 14:52:12.653350 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 42d8c79e0f7a444c869378678344f2b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42d8c79e0f7a444c869378678344f2b7 already in progress: creating tablet
W20250115 14:52:12.653527 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 638507d08f654011bea2597c22a788a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 638507d08f654011bea2597c22a788a2 already in progress: creating tablet
W20250115 14:52:12.653690 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 36de948083a946eba1a96f276e4f4f91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36de948083a946eba1a96f276e4f4f91 already in progress: creating tablet
W20250115 14:52:12.653878 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 288b0e455a854464940f65a2ab909d0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 288b0e455a854464940f65a2ab909d0b already in progress: creating tablet
W20250115 14:52:12.654052 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 36ac2c9c80824c25bb974092b3aba744 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36ac2c9c80824c25bb974092b3aba744 already in progress: creating tablet
W20250115 14:52:12.654230 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c9f67c7bfd4ab19a7a8bf385c9596b already in progress: creating tablet
W20250115 14:52:12.655776 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a696017e859f474bb1fd062df44c224d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a696017e859f474bb1fd062df44c224d already in progress: creating tablet
W20250115 14:52:12.656526 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c23b26e4b7ff4c699d1c86adf6731c71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c23b26e4b7ff4c699d1c86adf6731c71 already in progress: creating tablet
I20250115 14:52:12.657207 1167 tablet_replica.cc:331] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:12.657685 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7233ebf3bae1462d8eeb3669b39944c0
I20250115 14:52:12.658075 455 catalog_manager.cc:6596] ReplaceTablet: tablet 7233ebf3bae1462d8eeb3669b39944c0 deleted and replaced by tablet 2955ecdd45364dc2934ba96a9a6c8e8b
I20250115 14:52:12.658560 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 7233ebf3bae1462d8eeb3669b39944c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2955ecdd45364dc2934ba96a9a6c8e8b) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.659013 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7233ebf3bae1462d8eeb3669b39944c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2955ecdd45364dc2934ba96a9a6c8e8b) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.659231 1167 raft_consensus.cc:2238] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.659363 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 7233ebf3bae1462d8eeb3669b39944c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2955ecdd45364dc2934ba96a9a6c8e8b) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.660107 1751 ts_tablet_manager.cc:1918] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:12.660208 1751 log.cc:1198] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/c6f9262e808b4589a21257ed3fb0363b
I20250115 14:52:12.660566 1751 ts_tablet_manager.cc:1939] T c6f9262e808b4589a21257ed3fb0363b P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:12.660629 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf648bb656634def8189fb514a07e144 consensus state has no leader
I20250115 14:52:12.661326 1167 raft_consensus.cc:2267] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.662032 1164 tablet_bootstrap.cc:492] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:12.663225 1167 ts_tablet_manager.cc:1905] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.663410 1164 ts_tablet_manager.cc:1397] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.103s user 0.008s sys 0.000s
I20250115 14:52:12.664206 1164 raft_consensus.cc:357] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.664436 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1026) took 2005 ms. Trace:
I20250115 14:52:12.664528 1751 rpcz_store.cc:276] 0115 14:52:10.659031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.659088 (+ 57us) service_pool.cc:225] Handling call
0115 14:52:12.615318 (+1956230us) ts_tablet_manager.cc:1145] Deleting tablet c6f9262e808b4589a21257ed3fb0363b
0115 14:52:12.615326 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.643886 (+ 28560us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.659998 (+ 16112us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.664414 (+ 4416us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1956044}
I20250115 14:52:12.664932 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1025) took 2006 ms. Trace:
I20250115 14:52:12.664958 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet c6f9262e808b4589a21257ed3fb0363b (table default.Table_1da7201c533742a28cf8ad8777ecfb28 [id=eed10013d4554afbb09538836b5d3164]) successfully deleted
I20250115 14:52:12.665002 1751 rpcz_store.cc:276] 0115 14:52:10.658930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.659313 (+ 383us) service_pool.cc:225] Handling call
0115 14:52:12.664812 (+2005499us) ts_tablet_manager.cc:1145] Deleting tablet 9727430a15574ec0b24a5a64da744cac
0115 14:52:12.664823 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.664916 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2005332}
I20250115 14:52:12.665313 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1044) took 1807 ms. Trace:
W20250115 14:52:12.665395 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9727430a15574ec0b24a5a64da744cac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9727430a15574ec0b24a5a64da744cac already in progress: creating tablet
I20250115 14:52:12.665380 1751 rpcz_store.cc:276] 0115 14:52:10.857907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.858137 (+ 230us) service_pool.cc:225] Handling call
0115 14:52:12.665213 (+1807076us) ts_tablet_manager.cc:1145] Deleting tablet 5f0eeede8f714b92b1e3a9a558289c3e
0115 14:52:12.665220 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.665300 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806915}
I20250115 14:52:12.665653 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1047) took 1766 ms. Trace:
I20250115 14:52:12.665717 1751 rpcz_store.cc:276] 0115 14:52:10.898707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.899075 (+ 368us) service_pool.cc:225] Handling call
0115 14:52:12.665566 (+1766491us) ts_tablet_manager.cc:1145] Deleting tablet 8d4c30bccb3549e6963dfbbecf68cbbe
0115 14:52:12.665573 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.665641 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1763173}
W20250115 14:52:12.665773 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 5f0eeede8f714b92b1e3a9a558289c3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f0eeede8f714b92b1e3a9a558289c3e already in progress: creating tablet
I20250115 14:52:12.666004 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1051) took 1683 ms. Trace:
I20250115 14:52:12.666079 1751 rpcz_store.cc:276] 0115 14:52:10.982228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.982362 (+ 134us) service_pool.cc:225] Handling call
0115 14:52:12.665917 (+1683555us) ts_tablet_manager.cc:1145] Deleting tablet 42d8c79e0f7a444c869378678344f2b7
0115 14:52:12.665926 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.665989 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1683375}
W20250115 14:52:12.666368 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 8d4c30bccb3549e6963dfbbecf68cbbe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d4c30bccb3549e6963dfbbecf68cbbe already in progress: creating tablet
I20250115 14:52:12.666380 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1050) took 1686 ms. Trace:
I20250115 14:52:12.666440 1751 rpcz_store.cc:276] 0115 14:52:10.980077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.982597 (+ 2520us) service_pool.cc:225] Handling call
0115 14:52:12.666278 (+1683681us) ts_tablet_manager.cc:1145] Deleting tablet 638507d08f654011bea2597c22a788a2
0115 14:52:12.666286 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.666367 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1683571}
I20250115 14:52:12.666672 1751 tablet_replica.cc:331] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:12.666751 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 42d8c79e0f7a444c869378678344f2b7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42d8c79e0f7a444c869378678344f2b7 already in progress: creating tablet
I20250115 14:52:12.666851 1751 raft_consensus.cc:2238] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.666986 1751 raft_consensus.cc:2267] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:12.666985 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 638507d08f654011bea2597c22a788a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 638507d08f654011bea2597c22a788a2 already in progress: creating tablet
I20250115 14:52:12.667469 1751 ts_tablet_manager.cc:1905] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.671460 1168 tablet_bootstrap.cc:492] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.671602 1168 ts_tablet_manager.cc:1397] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.174s user 0.006s sys 0.014s
I20250115 14:52:12.663125 1166 ts_tablet_manager.cc:1918] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.671617 1164 raft_consensus.cc:383] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.671804 1166 log.cc:1198] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ff9b57b8b9434f55ace5a4f47380df54
I20250115 14:52:12.671940 1164 raft_consensus.cc:738] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.672233 1166 ts_tablet_manager.cc:1939] T ff9b57b8b9434f55ace5a4f47380df54 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:12.672266 1164 consensus_queue.cc:260] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.672399 1168 raft_consensus.cc:357] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.672578 1168 raft_consensus.cc:383] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.672647 1168 raft_consensus.cc:738] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.672852 1168 consensus_queue.cc:260] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.673338 1168 ts_tablet_manager.cc:1428] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:12.673561 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ff9b57b8b9434f55ace5a4f47380df54 (table default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]) successfully deleted
I20250115 14:52:12.673615 1166 tablet_replica.cc:331] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:12.673621 1168 tablet_bootstrap.cc:492] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.673759 1166 raft_consensus.cc:2238] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:12.673799 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4e39e13fb8c84ae18bac5e45d576a0ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e39e13fb8c84ae18bac5e45d576a0ab already in progress: creating tablet
I20250115 14:52:12.673905 1166 raft_consensus.cc:2267] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.674373 1166 ts_tablet_manager.cc:1905] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.675102 1168 tablet_bootstrap.cc:654] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.675606 1164 ts_tablet_manager.cc:1428] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250115 14:52:12.675994 1164 tablet_bootstrap.cc:492] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:12.678208 1751 ts_tablet_manager.cc:1918] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.678310 1751 log.cc:1198] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/b76d1bbf90c6416ca60805c9d8053bd4
I20250115 14:52:12.678565 1751 ts_tablet_manager.cc:1939] T b76d1bbf90c6416ca60805c9d8053bd4 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:12.679456 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-task__Z8ae/build/asan/bin/kudu
/tmp/dist-test-task__Z8ae/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40227
--webserver_interface=127.0.104.60
--webserver_port=37445
--builtin_ntp_servers=127.0.104.20:39931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250115 14:52:12.679823 1166 ts_tablet_manager.cc:1918] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:12.679908 1166 log.cc:1198] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/7233ebf3bae1462d8eeb3669b39944c0
I20250115 14:52:12.680212 1164 tablet_bootstrap.cc:654] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.680226 1166 ts_tablet_manager.cc:1939] T 7233ebf3bae1462d8eeb3669b39944c0 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:12.680931 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1049) took 1723 ms. Trace:
I20250115 14:52:12.681099 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_6a4b8c6d8e4e404981d168241fa84606 [id=8170a41205004f9ca83aa1c0ab7a2502]) successfully deleted
I20250115 14:52:12.681268 1751 rpcz_store.cc:276] 0115 14:52:10.957336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:10.984397 (+ 27061us) service_pool.cc:225] Handling call
0115 14:52:12.666632 (+1682235us) ts_tablet_manager.cc:1145] Deleting tablet b76d1bbf90c6416ca60805c9d8053bd4
0115 14:52:12.666639 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.675436 (+ 8797us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.678123 (+ 2687us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.680912 (+ 2789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682072}
I20250115 14:52:12.681661 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1052) took 1670 ms. Trace:
I20250115 14:52:12.681679 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet b76d1bbf90c6416ca60805c9d8053bd4 (table default.Table_9edd581e8fc4457697465360c4504ff1 [id=0ef6f07cf29e46e69bf2a4449187a4e7]) successfully deleted
I20250115 14:52:12.681728 1751 rpcz_store.cc:276] 0115 14:52:11.010979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.017286 (+ 6307us) service_pool.cc:225] Handling call
0115 14:52:12.681541 (+1664255us) ts_tablet_manager.cc:1145] Deleting tablet 36de948083a946eba1a96f276e4f4f91
0115 14:52:12.681551 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.681647 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1644173}
I20250115 14:52:12.682027 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1053) took 1392 ms. Trace:
W20250115 14:52:12.682114 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 36de948083a946eba1a96f276e4f4f91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36de948083a946eba1a96f276e4f4f91 already in progress: creating tablet
I20250115 14:52:12.682085 1751 rpcz_store.cc:276] 0115 14:52:11.289745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.289812 (+ 67us) service_pool.cc:225] Handling call
0115 14:52:12.681944 (+1392132us) ts_tablet_manager.cc:1145] Deleting tablet 288b0e455a854464940f65a2ab909d0b
0115 14:52:12.681952 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.682016 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1391968}
I20250115 14:52:12.682296 1751 tablet_replica.cc:331] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:12.682417 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 288b0e455a854464940f65a2ab909d0b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 288b0e455a854464940f65a2ab909d0b already in progress: creating tablet
I20250115 14:52:12.682457 1751 raft_consensus.cc:2238] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.682580 1751 raft_consensus.cc:2267] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.683058 1751 ts_tablet_manager.cc:1905] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.686884 1167 ts_tablet_manager.cc:1918] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.687001 1167 log.cc:1198] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/c37f4c2ab9d248a1af817cfa2c2e2ff0
I20250115 14:52:12.687254 1167 ts_tablet_manager.cc:1939] T c37f4c2ab9d248a1af817cfa2c2e2ff0 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:12.687605 1167 tablet_replica.cc:331] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:12.687800 1167 raft_consensus.cc:2238] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.687948 1167 raft_consensus.cc:2267] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.688448 1167 ts_tablet_manager.cc:1905] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.691100 1751 ts_tablet_manager.cc:1918] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.691211 1751 log.cc:1198] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/325cd3fd8b4d447998822163e575e758
I20250115 14:52:12.691481 1751 ts_tablet_manager.cc:1939] T 325cd3fd8b4d447998822163e575e758 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:12.691963 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet c37f4c2ab9d248a1af817cfa2c2e2ff0 (table default.Table_6f0b6c724c1c43e7adf538ad7369b7a1 [id=9e8ce51e784e4bdbb3ac78cba10bfedf]) successfully deleted
I20250115 14:52:12.692191 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1054) took 1399 ms. Trace:
I20250115 14:52:12.692276 1751 rpcz_store.cc:276] 0115 14:52:11.292393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.292452 (+ 59us) service_pool.cc:225] Handling call
0115 14:52:12.682257 (+1389805us) ts_tablet_manager.cc:1145] Deleting tablet 325cd3fd8b4d447998822163e575e758
0115 14:52:12.682264 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.686251 (+ 3987us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.691020 (+ 4769us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.692172 (+ 1152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1389650}
I20250115 14:52:12.693351 1751 tablet_replica.cc:331] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:12.693536 1751 raft_consensus.cc:2238] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.693667 1751 raft_consensus.cc:2267] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.694136 1751 ts_tablet_manager.cc:1905] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.695192 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 325cd3fd8b4d447998822163e575e758 (table default.Table_52bf98f0d5b44013ba19e15f0fc24cce [id=c7768272a8d14ac0aba3f569f8c8f58e]) successfully deleted
W20250115 14:52:12.695403 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a696017e859f474bb1fd062df44c224d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a696017e859f474bb1fd062df44c224d already in progress: creating tablet
W20250115 14:52:12.695546 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 36ac2c9c80824c25bb974092b3aba744 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36ac2c9c80824c25bb974092b3aba744 already in progress: creating tablet
W20250115 14:52:12.695712 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c23b26e4b7ff4c699d1c86adf6731c71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c23b26e4b7ff4c699d1c86adf6731c71 already in progress: creating tablet
W20250115 14:52:12.695880 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 33c9f67c7bfd4ab19a7a8bf385c9596b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33c9f67c7bfd4ab19a7a8bf385c9596b already in progress: creating tablet
W20250115 14:52:12.696058 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 3be094dcd3cb4f9d9259d354300c1600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3be094dcd3cb4f9d9259d354300c1600 already in progress: creating tablet
W20250115 14:52:12.696223 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ad417f06081d41248dff6f1eed175f29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad417f06081d41248dff6f1eed175f29 already in progress: creating tablet
W20250115 14:52:12.696420 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c6cb8dfa7c42480bbd139d5acf059a57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6cb8dfa7c42480bbd139d5acf059a57 already in progress: creating tablet
I20250115 14:52:12.698971 1164 tablet_bootstrap.cc:492] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:12.699105 1164 ts_tablet_manager.cc:1397] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20250115 14:52:12.699419 1162 tablet_bootstrap.cc:492] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.700223 1164 raft_consensus.cc:357] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.700430 1164 raft_consensus.cc:383] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.700484 1164 raft_consensus.cc:738] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.700667 1164 consensus_queue.cc:260] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.701249 1164 ts_tablet_manager.cc:1428] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:12.701484 1164 tablet_bootstrap.cc:492] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:12.701679 452 catalog_manager.cc:5526] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:12.702116 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a6f3f8cae6244968927b864e7ce97565 (table default.Table_5d9f72946dd846b98daa1df0c392cb72 [id=02f9f84c80334c36801ef8e760120ab3]). Expected version 2 got 0
I20250115 14:52:12.702318 452 catalog_manager.cc:5526] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:12.703332 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 25813a873f7440189b537742484b288e (DEFAULT_TABLE table=default.Table_8f3885b7d49b4591b043b5d2520d5c6f [id=24ab3abae47e4da882e2b0a0a8856b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.704028 749 tablet_service.cc:1467] Processing CreateTablet for tablet 25813a873f7440189b537742484b288e (DEFAULT_TABLE table=default.Table_8f3885b7d49b4591b043b5d2520d5c6f [id=24ab3abae47e4da882e2b0a0a8856b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.704357 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25813a873f7440189b537742484b288e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.705080 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 1896104fdf474aecbbcb77f0cab09e36 (DEFAULT_TABLE table=default.Table_ff8ce351b9514d26933886794f108c09 [id=bc16a55a970e42208d022df7008393d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.705399 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 24729156769443f8926577bae7707847 (DEFAULT_TABLE table=default.Table_47fb50f420d34a0f946869232d283ecc [id=456b9a2dc7944855a158298de62c3f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.705675 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 3e834623f9c7482d965afd23c89a2510 (DEFAULT_TABLE table=default.Table_442bce8b3c9842cfba4b7f94c280eb3e [id=e6eefa19b9d949d18862c4906140fe5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.706380 737 tablet_service.cc:1467] Processing CreateTablet for tablet 3e834623f9c7482d965afd23c89a2510 (DEFAULT_TABLE table=default.Table_442bce8b3c9842cfba4b7f94c280eb3e [id=e6eefa19b9d949d18862c4906140fe5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.706676 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e834623f9c7482d965afd23c89a2510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.709070 1751 ts_tablet_manager.cc:1918] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.709197 1751 log.cc:1198] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9f4475441e4a467a8d593b742d2a5b80
I20250115 14:52:12.709507 1751 ts_tablet_manager.cc:1939] T 9f4475441e4a467a8d593b742d2a5b80 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:12.710529 744 tablet_service.cc:1467] Processing CreateTablet for tablet eb2a6e23cafa4ce99514e063c8084cb4 (DEFAULT_TABLE table=default.Table_4f994af897944da0b4e1f9cf4a90e137 [id=4703d5b29fd4417092e65dd401b94b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.710842 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb2a6e23cafa4ce99514e063c8084cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.712563 755 tablet_service.cc:1467] Processing CreateTablet for tablet 24729156769443f8926577bae7707847 (DEFAULT_TABLE table=default.Table_47fb50f420d34a0f946869232d283ecc [id=456b9a2dc7944855a158298de62c3f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.712868 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24729156769443f8926577bae7707847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.714231 1014 tablet_service.cc:1467] Processing CreateTablet for tablet eb2a6e23cafa4ce99514e063c8084cb4 (DEFAULT_TABLE table=default.Table_4f994af897944da0b4e1f9cf4a90e137 [id=4703d5b29fd4417092e65dd401b94b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.714562 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 87b28e83081a4f6e959c8b26c68b81ce (DEFAULT_TABLE table=default.Table_da940b0d8b7d4e409e73adab5c24ba76 [id=8b81e10327fb453e870ad15f68431fa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.715312 747 tablet_service.cc:1467] Processing CreateTablet for tablet 1838833927a34258b9107c5e41212e18 (DEFAULT_TABLE table=default.Table_c172b6003608480ca15c2e050f16e291 [id=88f854c8c0f34ab186225dfaf0c718ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.715526 740 tablet_service.cc:1467] Processing CreateTablet for tablet d13576f09c8f4fcd906b696b4811b48a (DEFAULT_TABLE table=default.Table_b2a9e7fffbe84c459c9b07db2c59dca0 [id=0b84e386fb8a4b11a427a68e8b679b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.715633 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1838833927a34258b9107c5e41212e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.715818 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13576f09c8f4fcd906b696b4811b48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.716150 745 tablet_service.cc:1467] Processing CreateTablet for tablet abe3c66176664fa99c7a1fb480b56cca (DEFAULT_TABLE table=default.Table_02dcf1af493a4be5857c31c1c50836a7 [id=e9dbf1dbf63741e2b7f8a051bead5973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.716203 751 tablet_service.cc:1467] Processing CreateTablet for tablet 2f430135d6eb4484af60ee4425c3981c (DEFAULT_TABLE table=default.Table_4a036dcd4a7a41fab5a082306bba837b [id=00f6bc60c6fd4cdaa9e692c66b395b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.716482 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abe3c66176664fa99c7a1fb480b56cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.716722 739 tablet_service.cc:1467] Processing CreateTablet for tablet b3daf6416ad847aeaa83087cc81c9239 (DEFAULT_TABLE table=default.Table_d050f7c481a549248abc7c8beb08de6e [id=387d14a5bd4143bfbcac7e58f2c37511]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.716989 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3daf6416ad847aeaa83087cc81c9239. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.718580 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 2f430135d6eb4484af60ee4425c3981c (DEFAULT_TABLE table=default.Table_4a036dcd4a7a41fab5a082306bba837b [id=00f6bc60c6fd4cdaa9e692c66b395b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.718897 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1838833927a34258b9107c5e41212e18 (DEFAULT_TABLE table=default.Table_c172b6003608480ca15c2e050f16e291 [id=88f854c8c0f34ab186225dfaf0c718ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.719201 1018 tablet_service.cc:1467] Processing CreateTablet for tablet abe3c66176664fa99c7a1fb480b56cca (DEFAULT_TABLE table=default.Table_02dcf1af493a4be5857c31c1c50836a7 [id=e9dbf1dbf63741e2b7f8a051bead5973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.719388 742 tablet_service.cc:1467] Processing CreateTablet for tablet bc06a77162a6430bbbda7a3ca2c4e9ab (DEFAULT_TABLE table=default.Table_95f70e728c46403fa5002e0f2a01b5b8 [id=d297a568d2e4446e9a71d9147eeac913]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.719513 1022 tablet_service.cc:1467] Processing CreateTablet for tablet c497fb29752d4ed7bda207301f90cf05 (DEFAULT_TABLE table=default.Table_f8f4fd3823584b899089dc4b570b2756 [id=e2a2b0a8b2da47b2b45cfc63871e885b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.719651 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc06a77162a6430bbbda7a3ca2c4e9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.719703 752 tablet_service.cc:1467] Processing CreateTablet for tablet 70167f772ef246f2b1b5fdb380d6f4da (DEFAULT_TABLE table=default.Table_7a882c6c77fd4d198c18380628c8ef1a [id=630335788e2743a690d4879739aad240]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.719805 1016 tablet_service.cc:1467] Processing CreateTablet for tablet d13576f09c8f4fcd906b696b4811b48a (DEFAULT_TABLE table=default.Table_b2a9e7fffbe84c459c9b07db2c59dca0 [id=0b84e386fb8a4b11a427a68e8b679b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.719983 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70167f772ef246f2b1b5fdb380d6f4da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.720108 1028 tablet_service.cc:1467] Processing CreateTablet for tablet b3daf6416ad847aeaa83087cc81c9239 (DEFAULT_TABLE table=default.Table_d050f7c481a549248abc7c8beb08de6e [id=387d14a5bd4143bfbcac7e58f2c37511]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.720572 756 tablet_service.cc:1467] Processing CreateTablet for tablet c497fb29752d4ed7bda207301f90cf05 (DEFAULT_TABLE table=default.Table_f8f4fd3823584b899089dc4b570b2756 [id=e2a2b0a8b2da47b2b45cfc63871e885b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.720729 743 tablet_service.cc:1467] Processing CreateTablet for tablet c3d882a2515d4f8aa7f4fd7aa776f4e4 (DEFAULT_TABLE table=default.Table_39f9c4e24e5a4e54bde4760f73b4ff65 [id=5ee64c7e73ba4a72ab097b8c68fb7c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.720840 748 tablet_service.cc:1467] Processing CreateTablet for tablet 87b28e83081a4f6e959c8b26c68b81ce (DEFAULT_TABLE table=default.Table_da940b0d8b7d4e409e73adab5c24ba76 [id=8b81e10327fb453e870ad15f68431fa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.721030 741 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8e4992fc194f47836a1ea82b338cb4 (DEFAULT_TABLE table=default.Table_3c27ad12ca5f4ef3b72169e93eaf971f [id=31a62701669c437b946ab786b82699b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.716150 738 tablet_service.cc:1467] Processing CreateTablet for tablet 135ccb0e71f048f3ab19fd0faaea993a (DEFAULT_TABLE table=default.Table_fa6bcdff5ed349439e0b9e997ed620cf [id=8b64ae1a61e940028ab4611eaceb5dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.721290 1162 tablet_bootstrap.cc:654] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.721385 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f430135d6eb4484af60ee4425c3981c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.721688 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 70167f772ef246f2b1b5fdb380d6f4da (DEFAULT_TABLE table=default.Table_7a882c6c77fd4d198c18380628c8ef1a [id=630335788e2743a690d4879739aad240]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.703999 753 tablet_service.cc:1467] Processing CreateTablet for tablet 1896104fdf474aecbbcb77f0cab09e36 (DEFAULT_TABLE table=default.Table_ff8ce351b9514d26933886794f108c09 [id=bc16a55a970e42208d022df7008393d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.723693 1011 tablet_service.cc:1467] Processing CreateTablet for tablet bc06a77162a6430bbbda7a3ca2c4e9ab (DEFAULT_TABLE table=default.Table_95f70e728c46403fa5002e0f2a01b5b8 [id=d297a568d2e4446e9a71d9147eeac913]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.724036 1015 tablet_service.cc:1467] Processing CreateTablet for tablet c3d882a2515d4f8aa7f4fd7aa776f4e4 (DEFAULT_TABLE table=default.Table_39f9c4e24e5a4e54bde4760f73b4ff65 [id=5ee64c7e73ba4a72ab097b8c68fb7c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.724382 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8e4992fc194f47836a1ea82b338cb4 (DEFAULT_TABLE table=default.Table_3c27ad12ca5f4ef3b72169e93eaf971f [id=31a62701669c437b946ab786b82699b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.725015 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 05bb99f36ba649d8b607e9a95ab58ed0 (DEFAULT_TABLE table=default.Table_b053773cca094e6499791ad922f1fe10 [id=bd973eb84bcb426fa3e5963b7cb949dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.727818 1025 tablet_service.cc:1467] Processing CreateTablet for tablet c193e2aae66b4e53a1bd26d6004d34ba (DEFAULT_TABLE table=default.Table_662b6dfe0baa4437babff0046306d1dd [id=5d9a88eda7c94f358b4a5dfb33764308]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.724718 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 135ccb0e71f048f3ab19fd0faaea993a (DEFAULT_TABLE table=default.Table_fa6bcdff5ed349439e0b9e997ed620cf [id=8b64ae1a61e940028ab4611eaceb5dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.728231 1019 tablet_service.cc:1467] Processing CreateTablet for tablet c579a234cd814284a7837436ef357dfb (DEFAULT_TABLE table=default.Table_42e1685088164128aefa917c691bbf7b [id=0a1386a111224eb2b24f22584fa131ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.729766 1168 tablet_bootstrap.cc:492] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.729930 1168 ts_tablet_manager.cc:1397] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.000s
I20250115 14:52:12.730391 754 tablet_service.cc:1467] Processing CreateTablet for tablet 05bb99f36ba649d8b607e9a95ab58ed0 (DEFAULT_TABLE table=default.Table_b053773cca094e6499791ad922f1fe10 [id=bd973eb84bcb426fa3e5963b7cb949dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.730572 750 tablet_service.cc:1467] Processing CreateTablet for tablet c579a234cd814284a7837436ef357dfb (DEFAULT_TABLE table=default.Table_42e1685088164128aefa917c691bbf7b [id=0a1386a111224eb2b24f22584fa131ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.731123 746 tablet_service.cc:1467] Processing CreateTablet for tablet c193e2aae66b4e53a1bd26d6004d34ba (DEFAULT_TABLE table=default.Table_662b6dfe0baa4437babff0046306d1dd [id=5d9a88eda7c94f358b4a5dfb33764308]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.731165 1168 raft_consensus.cc:357] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.731436 1168 raft_consensus.cc:383] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.731519 1168 raft_consensus.cc:738] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.731772 1168 consensus_queue.cc:260] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.732414 1168 ts_tablet_manager.cc:1428] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:12.732686 1168 tablet_bootstrap.cc:492] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.734141 1168 tablet_bootstrap.cc:654] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.742704 874 tablet_service.cc:1467] Processing CreateTablet for tablet 24729156769443f8926577bae7707847 (DEFAULT_TABLE table=default.Table_47fb50f420d34a0f946869232d283ecc [id=456b9a2dc7944855a158298de62c3f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.742892 1164 tablet_bootstrap.cc:654] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.743016 891 tablet_service.cc:1467] Processing CreateTablet for tablet 1896104fdf474aecbbcb77f0cab09e36 (DEFAULT_TABLE table=default.Table_ff8ce351b9514d26933886794f108c09 [id=bc16a55a970e42208d022df7008393d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.743106 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24729156769443f8926577bae7707847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.743407 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1896104fdf474aecbbcb77f0cab09e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.746193 1167 ts_tablet_manager.cc:1918] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.746304 1167 log.cc:1198] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ff9b57b8b9434f55ace5a4f47380df54
I20250115 14:52:12.746603 1167 ts_tablet_manager.cc:1939] T ff9b57b8b9434f55ace5a4f47380df54 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:12.747001 1167 tablet_replica.cc:331] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:12.747195 1167 raft_consensus.cc:2238] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.747349 1167 raft_consensus.cc:2267] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.747838 1167 ts_tablet_manager.cc:1905] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.742666 883 tablet_service.cc:1467] Processing CreateTablet for tablet 25813a873f7440189b537742484b288e (DEFAULT_TABLE table=default.Table_8f3885b7d49b4591b043b5d2520d5c6f [id=24ab3abae47e4da882e2b0a0a8856b53]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.748346 2549 raft_consensus.cc:491] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.748569 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25813a873f7440189b537742484b288e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.748471 2549 raft_consensus.cc:513] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.749117 2549 leader_election.cc:290] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:12.749075 893 tablet_service.cc:1467] Processing CreateTablet for tablet 1838833927a34258b9107c5e41212e18 (DEFAULT_TABLE table=default.Table_c172b6003608480ca15c2e050f16e291 [id=88f854c8c0f34ab186225dfaf0c718ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.749402 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3e834623f9c7482d965afd23c89a2510 (DEFAULT_TABLE table=default.Table_442bce8b3c9842cfba4b7f94c280eb3e [id=e6eefa19b9d949d18862c4906140fe5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.749862 882 tablet_service.cc:1467] Processing CreateTablet for tablet eb2a6e23cafa4ce99514e063c8084cb4 (DEFAULT_TABLE table=default.Table_4f994af897944da0b4e1f9cf4a90e137 [id=4703d5b29fd4417092e65dd401b94b15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.750036 884 tablet_service.cc:1467] Processing CreateTablet for tablet abe3c66176664fa99c7a1fb480b56cca (DEFAULT_TABLE table=default.Table_02dcf1af493a4be5857c31c1c50836a7 [id=e9dbf1dbf63741e2b7f8a051bead5973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.750301 878 tablet_service.cc:1467] Processing CreateTablet for tablet 2f430135d6eb4484af60ee4425c3981c (DEFAULT_TABLE table=default.Table_4a036dcd4a7a41fab5a082306bba837b [id=00f6bc60c6fd4cdaa9e692c66b395b56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.750787 890 tablet_service.cc:1467] Processing CreateTablet for tablet c497fb29752d4ed7bda207301f90cf05 (DEFAULT_TABLE table=default.Table_f8f4fd3823584b899089dc4b570b2756 [id=e2a2b0a8b2da47b2b45cfc63871e885b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.751511 892 tablet_service.cc:1467] Processing CreateTablet for tablet d13576f09c8f4fcd906b696b4811b48a (DEFAULT_TABLE table=default.Table_b2a9e7fffbe84c459c9b07db2c59dca0 [id=0b84e386fb8a4b11a427a68e8b679b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.751803 876 tablet_service.cc:1467] Processing CreateTablet for tablet b3daf6416ad847aeaa83087cc81c9239 (DEFAULT_TABLE table=default.Table_d050f7c481a549248abc7c8beb08de6e [id=387d14a5bd4143bfbcac7e58f2c37511]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.751978 889 tablet_service.cc:1467] Processing CreateTablet for tablet 87b28e83081a4f6e959c8b26c68b81ce (DEFAULT_TABLE table=default.Table_da940b0d8b7d4e409e73adab5c24ba76 [id=8b81e10327fb453e870ad15f68431fa1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:12.752526 847 leader_election.cc:343] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 4294c49da1e044be993100b3cf5e2cea
I20250115 14:52:12.749408 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1838833927a34258b9107c5e41212e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.752772 886 tablet_service.cc:1467] Processing CreateTablet for tablet 70167f772ef246f2b1b5fdb380d6f4da (DEFAULT_TABLE table=default.Table_7a882c6c77fd4d198c18380628c8ef1a [id=630335788e2743a690d4879739aad240]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.752853 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3daf6416ad847aeaa83087cc81c9239. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.753036 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb2a6e23cafa4ce99514e063c8084cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.753192 881 tablet_service.cc:1467] Processing CreateTablet for tablet bc06a77162a6430bbbda7a3ca2c4e9ab (DEFAULT_TABLE table=default.Table_95f70e728c46403fa5002e0f2a01b5b8 [id=d297a568d2e4446e9a71d9147eeac913]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.753396 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c497fb29752d4ed7bda207301f90cf05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.753556 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc06a77162a6430bbbda7a3ca2c4e9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.756397 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70167f772ef246f2b1b5fdb380d6f4da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.756559 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abe3c66176664fa99c7a1fb480b56cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.756954 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e834623f9c7482d965afd23c89a2510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.759270 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13576f09c8f4fcd906b696b4811b48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.760408 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f430135d6eb4484af60ee4425c3981c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.764401 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b28e83081a4f6e959c8b26c68b81ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.776510 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d882a2515d4f8aa7f4fd7aa776f4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.776259 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25813a873f7440189b537742484b288e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.781054 881 tablet_service.cc:1467] Processing CreateTablet for tablet 135ccb0e71f048f3ab19fd0faaea993a (DEFAULT_TABLE table=default.Table_fa6bcdff5ed349439e0b9e997ed620cf [id=8b64ae1a61e940028ab4611eaceb5dc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.781528 876 tablet_service.cc:1467] Processing CreateTablet for tablet c579a234cd814284a7837436ef357dfb (DEFAULT_TABLE table=default.Table_42e1685088164128aefa917c691bbf7b [id=0a1386a111224eb2b24f22584fa131ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.781728 889 tablet_service.cc:1467] Processing CreateTablet for tablet c3d882a2515d4f8aa7f4fd7aa776f4e4 (DEFAULT_TABLE table=default.Table_39f9c4e24e5a4e54bde4760f73b4ff65 [id=5ee64c7e73ba4a72ab097b8c68fb7c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.781940 884 tablet_service.cc:1467] Processing CreateTablet for tablet c193e2aae66b4e53a1bd26d6004d34ba (DEFAULT_TABLE table=default.Table_662b6dfe0baa4437babff0046306d1dd [id=5d9a88eda7c94f358b4a5dfb33764308]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.782200 887 tablet_service.cc:1467] Processing CreateTablet for tablet 05bb99f36ba649d8b607e9a95ab58ed0 (DEFAULT_TABLE table=default.Table_b053773cca094e6499791ad922f1fe10 [id=bd973eb84bcb426fa3e5963b7cb949dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.782384 890 tablet_service.cc:1467] Processing CreateTablet for tablet 482c5a8deb23418587bb333b3615eeea (DEFAULT_TABLE table=default.Table_7a7640e000334453a1519964f3fee50c [id=3b2cfeb56818485489aa32baa1a79c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.782673 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 135ccb0e71f048f3ab19fd0faaea993a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.782977 878 tablet_service.cc:1467] Processing CreateTablet for tablet 0f8e4992fc194f47836a1ea82b338cb4 (DEFAULT_TABLE table=default.Table_3c27ad12ca5f4ef3b72169e93eaf971f [id=31a62701669c437b946ab786b82699b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.783262 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f8e4992fc194f47836a1ea82b338cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.784724 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet ff9b57b8b9434f55ace5a4f47380df54 (table default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]) successfully deleted
I20250115 14:52:12.785851 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bb99f36ba649d8b607e9a95ab58ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.786764 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d882a2515d4f8aa7f4fd7aa776f4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.788398 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c193e2aae66b4e53a1bd26d6004d34ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.788321 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4294c49da1e044be993100b3cf5e2cea" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.792397 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c579a234cd814284a7837436ef357dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.796387 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 482c5a8deb23418587bb333b3615eeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.800887 743 tablet_service.cc:1467] Processing CreateTablet for tablet 482c5a8deb23418587bb333b3615eeea (DEFAULT_TABLE table=default.Table_7a7640e000334453a1519964f3fee50c [id=3b2cfeb56818485489aa32baa1a79c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.802973 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 482c5a8deb23418587bb333b3615eeea. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:12.803354 848 leader_election.cc:343] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.803474 848 leader_election.cc:304] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.808543 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 6afbdd0b585e4e7e82067dce061924d0
I20250115 14:52:12.809017 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1896104fdf474aecbbcb77f0cab09e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.809329 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 856533ed254a49b4a461ed1ad3eb405e
I20250115 14:52:12.809723 451 catalog_manager.cc:6596] ReplaceTablet: tablet 856533ed254a49b4a461ed1ad3eb405e deleted and replaced by tablet e6821f4c34f042c8b4819176fe4418ff
I20250115 14:52:12.812480 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24729156769443f8926577bae7707847. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.813371 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a6f3f8cae6244968927b864e7ce97565 (table default.Table_5d9f72946dd846b98daa1df0c392cb72 [id=02f9f84c80334c36801ef8e760120ab3]). Expected version 2 got 0
I20250115 14:52:12.813599 448 catalog_manager.cc:5526] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:12.814097 448 catalog_manager.cc:5526] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:12.816473 2341 raft_consensus.cc:2743] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.817113 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e834623f9c7482d965afd23c89a2510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.817878 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_54e0718fb03d45629ac03649aacf44ba" } modify_external_catalogs: true
I20250115 14:52:12.818042 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_54e0718fb03d45629ac03649aacf44ba" } modify_external_catalogs: true
I20250115 14:52:12.818544 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_28327356092a4fd1b277babda6f161dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.819008 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28327356092a4fd1b277babda6f161dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.820080 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e3e5361d9cb144999d02ff4df3e1999b" } modify_external_catalogs: true
I20250115 14:52:12.820283 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 6afbdd0b585e4e7e82067dce061924d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.820278 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e3e5361d9cb144999d02ff4df3e1999b" } modify_external_catalogs: true
I20250115 14:52:12.820513 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 856533ed254a49b4a461ed1ad3eb405e with delete_type TABLET_DATA_DELETED (replaced by tablet e6821f4c34f042c8b4819176fe4418ff) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.820912 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d47eeb687f8046f2803dd6b6a5e6c153"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.821377 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d47eeb687f8046f2803dd6b6a5e6c153 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.824381 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 482c5a8deb23418587bb333b3615eeea (DEFAULT_TABLE table=default.Table_7a7640e000334453a1519964f3fee50c [id=3b2cfeb56818485489aa32baa1a79c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.824710 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0034e4c667b44bbb83a5a93e321e5113" } modify_external_catalogs: true
I20250115 14:52:12.824716 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 482c5a8deb23418587bb333b3615eeea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.824867 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0034e4c667b44bbb83a5a93e321e5113" } modify_external_catalogs: true
I20250115 14:52:12.826220 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_10df3b88664a48a28d08136f0c29a7e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.826658 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10df3b88664a48a28d08136f0c29a7e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.827667 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0218bb8722464e328f7582eacf088585" } modify_external_catalogs: true
I20250115 14:52:12.827842 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0218bb8722464e328f7582eacf088585" } modify_external_catalogs: true
I20250115 14:52:12.829146 1168 tablet_bootstrap.cc:492] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:12.829296 1168 ts_tablet_manager.cc:1397] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.001s
I20250115 14:52:12.829419 455 master_service.cc:946] ReplaceTablet: received request to replace tablet ac97cd8e11cf4b5e8978902d1c99aad8 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:12.830541 1162 tablet_bootstrap.cc:492] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:12.830653 1162 ts_tablet_manager.cc:1397] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.131s user 0.007s sys 0.003s
I20250115 14:52:12.830682 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 6afbdd0b585e4e7e82067dce061924d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.830865 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 856533ed254a49b4a461ed1ad3eb405e with delete_type TABLET_DATA_DELETED (replaced by tablet e6821f4c34f042c8b4819176fe4418ff) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.830974 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd94e4a7fef44f5be462e7a77e8a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.831077 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cae4c9228c640e495507e8f7ad9f544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.831339 1162 raft_consensus.cc:357] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.831519 1162 raft_consensus.cc:383] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.831568 1162 raft_consensus.cc:738] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.831874 1162 consensus_queue.cc:260] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.833463 1162 ts_tablet_manager.cc:1428] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:12.834004 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 6afbdd0b585e4e7e82067dce061924d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.834087 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 856533ed254a49b4a461ed1ad3eb405e with delete_type TABLET_DATA_DELETED (replaced by tablet e6821f4c34f042c8b4819176fe4418ff) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.834192 740 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2d04209fd34918b83978325a5e1df8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.834192 749 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1ce59e05054a009cd36e12db35d449 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.834713 1166 tablet_replica.cc:331] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:12.834893 1166 raft_consensus.cc:2238] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.835053 1166 raft_consensus.cc:2267] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.835839 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c497fb29752d4ed7bda207301f90cf05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.835891 1166 ts_tablet_manager.cc:1905] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:12.837224 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 6afbdd0b585e4e7e82067dce061924d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6afbdd0b585e4e7e82067dce061924d0 already in progress: creating tablet
W20250115 14:52:12.837445 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 856533ed254a49b4a461ed1ad3eb405e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 856533ed254a49b4a461ed1ad3eb405e already in progress: creating tablet
I20250115 14:52:12.840509 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c579a234cd814284a7837436ef357dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.840965 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb2a6e23cafa4ce99514e063c8084cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.841964 2563 consensus_queue.cc:1035] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:12.842332 2563 consensus_queue.cc:1035] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:12.844512 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b28e83081a4f6e959c8b26c68b81ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.844892 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f8e4992fc194f47836a1ea82b338cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.848526 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f430135d6eb4484af60ee4425c3981c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.852324 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b28e83081a4f6e959c8b26c68b81ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.852525 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1896104fdf474aecbbcb77f0cab09e36. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.855229 1166 ts_tablet_manager.cc:1918] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.855322 1166 log.cc:1198] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/1b2d04209fd34918b83978325a5e1df8
I20250115 14:52:12.855770 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bb99f36ba649d8b607e9a95ab58ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.855940 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 135ccb0e71f048f3ab19fd0faaea993a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.856058 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1838833927a34258b9107c5e41212e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.857081 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c193e2aae66b4e53a1bd26d6004d34ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.860477 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c193e2aae66b4e53a1bd26d6004d34ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.861315 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3d882a2515d4f8aa7f4fd7aa776f4e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.863454 1166 ts_tablet_manager.cc:1939] T 1b2d04209fd34918b83978325a5e1df8 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:12.864007 1162 tablet_bootstrap.cc:492] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:12.864487 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 135ccb0e71f048f3ab19fd0faaea993a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.864701 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f8e4992fc194f47836a1ea82b338cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.864876 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c579a234cd814284a7837436ef357dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.865025 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70167f772ef246f2b1b5fdb380d6f4da. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.868446 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c497fb29752d4ed7bda207301f90cf05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.868885 1168 raft_consensus.cc:357] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.869109 1168 raft_consensus.cc:383] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.869191 1168 raft_consensus.cc:738] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.869417 1168 consensus_queue.cc:260] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.869976 1168 ts_tablet_manager.cc:1428] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.041s user 0.001s sys 0.000s
I20250115 14:52:12.870111 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3daf6416ad847aeaa83087cc81c9239. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.870255 1168 tablet_bootstrap.cc:492] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:12.871592 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc06a77162a6430bbbda7a3ca2c4e9ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.872320 1162 tablet_bootstrap.cc:654] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.876394 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d13576f09c8f4fcd906b696b4811b48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.880403 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05bb99f36ba649d8b607e9a95ab58ed0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.884408 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abe3c66176664fa99c7a1fb480b56cca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.888497 2564 raft_consensus.cc:491] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.888628 2564 raft_consensus.cc:513] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.889110 1168 tablet_bootstrap.cc:654] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.889297 2564 leader_election.cc:290] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.893013 1166 tablet_replica.cc:331] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:12.893211 1166 raft_consensus.cc:2238] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:12.893451 1166 pending_rounds.cc:62] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: Trying to abort 1 pending ops.
I20250115 14:52:12.893487 1166 pending_rounds.cc:69] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558392933093376 op_type: NO_OP noop_request { }
I20250115 14:52:12.893595 1166 raft_consensus.cc:2883] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:12.893662 1166 raft_consensus.cc:2267] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.894516 1166 ts_tablet_manager.cc:1905] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.896687 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48a0af9422f43cf86531665c0fa1aee" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:12.898017 983 leader_election.cc:343] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.899601 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c5c33df1942c4b90a8069108477e91f5" } new_table_name: "default.Table_a14212e9310040fa88dd2432f80fdc58" modify_external_catalogs: true
I20250115 14:52:12.901108 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 97b599c6b8174959b8b66e90e9242057
I20250115 14:52:12.901973 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f4c2ee7746d24f588e97be9b31f6f62a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.902515 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4c2ee7746d24f588e97be9b31f6f62a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.902467 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6a4b8c6d8e4e404981d168241fa84606" } new_table_name: "default.Table_1b5ca8c3048846588478dbfbebf1ff82" modify_external_catalogs: true
I20250115 14:52:12.903057 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3991253a712b4579981713c7f964a053" } new_table_name: "default.Table_79a27d4c8b264b00b293966218eedf86" modify_external_catalogs: true
I20250115 14:52:12.905668 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b5937d6ad0494734a402fdc594eb9415 with delete_type TABLET_DATA_DELETED (replaced by tablet f2ba4086a6bf4bfa83e740b31dfb53f8) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.906599 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 742792be08444792892bd9cd7336dda3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.907155 1751 tablet_replica.cc:331] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:12.907354 1751 raft_consensus.cc:2238] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.907491 1751 raft_consensus.cc:2267] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.907999 1751 ts_tablet_manager.cc:1905] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.908630 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4294c49da1e044be993100b3cf5e2cea with delete_type TABLET_DATA_DELETED (replaced by tablet d1259256b7744e739581125611abbe67) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.909019 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b82924e735846b2914520684d1b8fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.909319 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ba596cfb3ce64718a42a9083975edc8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:12.910220 848 consensus_peers.cc:487] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:12.913640 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 1b2d04209fd34918b83978325a5e1df8 (table default.Table_d548be616a2c4f01ac52eadd5c009eaf [id=1907295f0b5c49b9b99a188d4dc9f7f8]) successfully deleted
I20250115 14:52:12.916384 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce498ffd3f04c5d8eb642739a838903 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.916671 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b599c6b8174959b8b66e90e9242057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.916903 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 65a6f83730f74d13823bd15d93538df7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.918604 1751 ts_tablet_manager.cc:1918] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.918758 1751 log.cc:1198] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/398ac8af619c427287b3964443173864
I20250115 14:52:12.919086 1751 ts_tablet_manager.cc:1939] T 398ac8af619c427287b3964443173864 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:12.924163 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9f4475441e4a467a8d593b742d2a5b80 (table default.Table_c763759bb01d449cbd6e2f851b303ec5 [id=4f4b4f8aa6364221b4a35c423a5631aa]) successfully deleted
I20250115 14:52:12.924520 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet cc9701156b0e48498666e34b3bec3043
I20250115 14:52:12.925166 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_49688b4238ca4abaac09f3cbdb738761" } new_table_name: "default.Table_963dec75332a458f8292d8dc2a1e2338" modify_external_catalogs: true
I20250115 14:52:12.926301 1167 ts_tablet_manager.cc:1918] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:12.926417 1167 log.cc:1198] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b5177265ba974ad6a381adb1e8c49589
I20250115 14:52:12.926502 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ac97cd8e11cf4b5e8978902d1c99aad8
I20250115 14:52:12.926729 1167 ts_tablet_manager.cc:1939] T b5177265ba974ad6a381adb1e8c49589 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:12.926982 455 catalog_manager.cc:6596] ReplaceTablet: tablet ac97cd8e11cf4b5e8978902d1c99aad8 deleted and replaced by tablet b5a476872b254f6989640209077bf38c
I20250115 14:52:12.927067 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1078) took 1079 ms. Trace:
I20250115 14:52:12.927152 1167 rpcz_store.cc:276] 0115 14:52:11.847644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.847765 (+ 121us) service_pool.cc:225] Handling call
0115 14:52:12.746939 (+899174us) ts_tablet_manager.cc:1145] Deleting tablet b5177265ba974ad6a381adb1e8c49589
0115 14:52:12.746945 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.750393 (+ 3448us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.926176 (+175783us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:12.927035 (+ 859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":899085}
I20250115 14:52:12.927469 1167 tablet_replica.cc:331] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:12.927392 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_db34dca850e84ecc810cb875e2b237cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:12.927657 1167 raft_consensus.cc:2238] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.927839 1167 raft_consensus.cc:2267] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:12.927945 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db34dca850e84ecc810cb875e2b237cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.928349 1167 ts_tablet_manager.cc:1905] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.928452 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a300a76b8f7942dea648a20cbb3d5368
I20250115 14:52:12.929101 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8db4c9b252ac4eaf90ab6b556fdbdefc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.929533 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8db4c9b252ac4eaf90ab6b556fdbdefc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.929828 2469 raft_consensus.cc:491] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.929947 2469 raft_consensus.cc:513] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.930536 2469 leader_election.cc:290] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.933434 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48a0af9422f43cf86531665c0fa1aee" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.924103 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_913e9383036c49f985fe9b1167c65efc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.935350 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_913e9383036c49f985fe9b1167c65efc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:12.936926 984 leader_election.cc:343] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.937115 984 leader_election.cc:304] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.939622 1751 tablet_replica.cc:331] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:12.939811 1751 raft_consensus.cc:2238] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:12.940119 2469 raft_consensus.cc:2743] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.940404 1751 raft_consensus.cc:2267] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:12.941094 738 tablet_service.cc:1514] Processing DeleteTablet for tablet a300a76b8f7942dea648a20cbb3d5368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.941684 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 398ac8af619c427287b3964443173864 (table default.Table_a26e212086b1478cb159097102934dd1 [id=4bb94d3cca0c4a14aad6957911139e51]) successfully deleted
I20250115 14:52:12.941790 743 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9701156b0e48498666e34b3bec3043 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.942773 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e244689cef6844e2935becd265eecad0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:12.942934 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b333d2c5a53745b9aeca2726318f628c consensus state has no leader
W20250115 14:52:12.943518 983 leader_election.cc:343] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.944563 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet ac97cd8e11cf4b5e8978902d1c99aad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b5a476872b254f6989640209077bf38c) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.944723 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9701156b0e48498666e34b3bec3043 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.944737 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3b9fd4f4e09a40e99f015f099eef8306" } new_table_name: "default.Table_ee5ad6bec8a445929d69cf0a962970d8" modify_external_catalogs: true
I20250115 14:52:12.945328 1164 tablet_bootstrap.cc:492] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:12.945439 1164 ts_tablet_manager.cc:1397] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.244s user 0.007s sys 0.000s
I20250115 14:52:12.946098 1164 raft_consensus.cc:357] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.946259 1164 raft_consensus.cc:383] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:12.946296 1164 raft_consensus.cc:738] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:12.946444 1164 consensus_queue.cc:260] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.946414 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_454909c032fc4a9b82dd07b728e001ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.946906 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_454909c032fc4a9b82dd07b728e001ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.946959 1164 ts_tablet_manager.cc:1428] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250115 14:52:12.947186 1164 tablet_bootstrap.cc:492] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:12.948505 1164 tablet_bootstrap.cc:654] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:12.949147 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet a300a76b8f7942dea648a20cbb3d5368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:12.949081 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_aa12615b9ec74e788e7883960ac22d2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.949529 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa12615b9ec74e788e7883960ac22d2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.950127 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b599c6b8174959b8b66e90e9242057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:12.950753 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2955ecdd45364dc2934ba96a9a6c8e8b has no consensus state
I20250115 14:52:12.951167 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_715e348f040b4062aeafb7c644b81772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:12.951629 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_715e348f040b4062aeafb7c644b81772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:12.952343 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e244689cef6844e2935becd265eecad0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:12.957096 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b599c6b8174959b8b66e90e9242057 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:12.957814 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ac97cd8e11cf4b5e8978902d1c99aad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b5a476872b254f6989640209077bf38c) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:12.958251 984 leader_election.cc:343] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.958369 984 leader_election.cc:304] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.958657 2469 raft_consensus.cc:2743] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.959200 1751 ts_tablet_manager.cc:1905] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:12.960976 2469 raft_consensus.cc:491] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.961110 2469 raft_consensus.cc:513] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:12.961706 2469 leader_election.cc:290] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:12.961928 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 026d05f8fb584da39000d83546b09e63 consensus state has no leader
I20250115 14:52:12.962396 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04daad461f0b433ba635df87439b5bc1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:12.962872 983 leader_election.cc:343] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.963549 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04daad461f0b433ba635df87439b5bc1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:12.964051 984 leader_election.cc:343] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:12.964190 984 leader_election.cc:304] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:12.964458 2469 raft_consensus.cc:2743] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:12.963822 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b5177265ba974ad6a381adb1e8c49589 (table default.Table_b6a520bf69264c44b1da30c0ffb41cc0 [id=f0b642abaec54506b98190fd1ac9ce67]) successfully deleted
W20250115 14:52:12.973883 848 consensus_peers.cc:487] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7233ebf3bae1462d8eeb3669b39944c0. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:12.982592 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3aa67705596742c6b942f49bca0aa817 consensus state has no leader
I20250115 14:52:12.983899 889 tablet_service.cc:1514] Processing DeleteTablet for tablet a300a76b8f7942dea648a20cbb3d5368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.984200 887 tablet_service.cc:1514] Processing DeleteTablet for tablet ac97cd8e11cf4b5e8978902d1c99aad8 with delete_type TABLET_DATA_DELETED (replaced by tablet b5a476872b254f6989640209077bf38c) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.984484 878 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9701156b0e48498666e34b3bec3043 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:12.989722 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 10c9de0f178743f88e1334c5397e4fd3 (DEFAULT_TABLE table=default.Table_3089c6f63edb4642b0ed0a82ee1b5051 [id=cf05513d06aa45289a106e93095873d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.990257 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10c9de0f178743f88e1334c5397e4fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.991338 450 catalog_manager.cc:5526] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:12.994159 878 tablet_service.cc:1467] Processing CreateTablet for tablet 10c9de0f178743f88e1334c5397e4fd3 (DEFAULT_TABLE table=default.Table_3089c6f63edb4642b0ed0a82ee1b5051 [id=cf05513d06aa45289a106e93095873d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.994588 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10c9de0f178743f88e1334c5397e4fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.996052 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 19154be028d44ba2bed0f54ae04cc770 (DEFAULT_TABLE table=default.Table_7c4783895e7c4f9da20efe43779ce1e0 [id=fbfbae91fdde4a529a4e4b35f4699ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.996233 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7d583cf1f44fe79397e238d5d0a66e (DEFAULT_TABLE table=default.Table_d5bce1994c5a4329bb30179b053babbb [id=a437c9a8578b4ba6a9b20a5de4e35af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.996417 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19154be028d44ba2bed0f54ae04cc770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.996665 1010 tablet_service.cc:1467] Processing CreateTablet for tablet d715a54286854b64bcf9535303460519 (DEFAULT_TABLE table=default.Table_9ab0ec2875a044198bb7906f36b7680b [id=6363c0b085dd4bf5923017a4007e9982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.996762 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 2955ecdd45364dc2934ba96a9a6c8e8b (DEFAULT_TABLE table=default.Table_1b5ca8c3048846588478dbfbebf1ff82 [id=8170a41205004f9ca83aa1c0ab7a2502]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.996945 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d715a54286854b64bcf9535303460519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.997100 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2955ecdd45364dc2934ba96a9a6c8e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.997269 2403 raft_consensus.cc:491] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:12.997398 2403 raft_consensus.cc:513] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:12.997927 2403 leader_election.cc:290] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:12.998158 754 tablet_service.cc:1467] Processing CreateTablet for tablet 1b44a481d86d48649c1a82a6b0ed9403 (DEFAULT_TABLE table=default.Table_ca37f38ca7de45edba3a3cbcfae56bd4 [id=3c524adfd92349df9b4f74ab5c41a1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.998203 755 tablet_service.cc:1467] Processing CreateTablet for tablet 2955ecdd45364dc2934ba96a9a6c8e8b (DEFAULT_TABLE table=default.Table_1b5ca8c3048846588478dbfbebf1ff82 [id=8170a41205004f9ca83aa1c0ab7a2502]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.998363 744 tablet_service.cc:1467] Processing CreateTablet for tablet a0d8d2e9c2c547debfb8ac410192a955 (DEFAULT_TABLE table=default.Table_4607c9a022124589b931dfbb075cefb5 [id=de749f3cbdc34b039fdb24c3927469d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.998529 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2955ecdd45364dc2934ba96a9a6c8e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.998660 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d8d2e9c2c547debfb8ac410192a955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:12.998673 738 tablet_service.cc:1467] Processing CreateTablet for tablet c487e956367845299e20cea05ce52987 (DEFAULT_TABLE table=default.Table_f4d5908220b548f1ba60227f435a948a [id=9e76d99bc062464580862b5b4bc023ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:12.999047 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c487e956367845299e20cea05ce52987. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.000519 1013 tablet_service.cc:1467] Processing CreateTablet for tablet a0d8d2e9c2c547debfb8ac410192a955 (DEFAULT_TABLE table=default.Table_4607c9a022124589b931dfbb075cefb5 [id=de749f3cbdc34b039fdb24c3927469d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.000800 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d8d2e9c2c547debfb8ac410192a955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.003934 753 tablet_service.cc:1467] Processing CreateTablet for tablet 69a5e5c5758445d3b97615e82f91fcf7 (DEFAULT_TABLE table=default.Table_ee7134670e574f279ad4f5557f395e69 [id=4d5a034e3cd94f3fbb85cb8aea5d4527]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.004217 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a5e5c5758445d3b97615e82f91fcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.004846 748 tablet_service.cc:1467] Processing CreateTablet for tablet 10c9de0f178743f88e1334c5397e4fd3 (DEFAULT_TABLE table=default.Table_3089c6f63edb4642b0ed0a82ee1b5051 [id=cf05513d06aa45289a106e93095873d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.005126 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10c9de0f178743f88e1334c5397e4fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.006857 746 tablet_service.cc:1467] Processing CreateTablet for tablet 2923dd60927d49f5b0ca09438cfd4ce3 (DEFAULT_TABLE table=default.Table_262d362a9eb447c689cbb66e877ab7be [id=7f7b3587d95c405ea2e217d74434180a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.006942 1021 tablet_service.cc:1467] Processing CreateTablet for tablet c487e956367845299e20cea05ce52987 (DEFAULT_TABLE table=default.Table_f4d5908220b548f1ba60227f435a948a [id=9e76d99bc062464580862b5b4bc023ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.007130 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2923dd60927d49f5b0ca09438cfd4ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.007223 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c487e956367845299e20cea05ce52987. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:13.008889 710 leader_election.cc:343] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 742792be08444792892bd9cd7336dda3
I20250115 14:52:13.012485 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 2923dd60927d49f5b0ca09438cfd4ce3 (DEFAULT_TABLE table=default.Table_262d362a9eb447c689cbb66e877ab7be [id=7f7b3587d95c405ea2e217d74434180a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.012794 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2923dd60927d49f5b0ca09438cfd4ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.013578 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b15de9bf8943cdb9e5abd943484993 with delete_type TABLET_DATA_DELETED (replaced by tablet 1de35551e26643b3b1e570ab8b2f6f56) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.014309 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b44a481d86d48649c1a82a6b0ed9403. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.015518 1016 tablet_service.cc:1467] Processing CreateTablet for tablet b4547b5be83e403db23a2ab92c7d0f6d (DEFAULT_TABLE table=default.Table_bc62cb62b8884088aa71a60746b07ffd [id=ad2bd57f7ddb42768c0f80fa87c516ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.015798 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4547b5be83e403db23a2ab92c7d0f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.019146 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 1b44a481d86d48649c1a82a6b0ed9403 (DEFAULT_TABLE table=default.Table_ca37f38ca7de45edba3a3cbcfae56bd4 [id=3c524adfd92349df9b4f74ab5c41a1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.019549 750 tablet_service.cc:1467] Processing CreateTablet for tablet d715a54286854b64bcf9535303460519 (DEFAULT_TABLE table=default.Table_9ab0ec2875a044198bb7906f36b7680b [id=6363c0b085dd4bf5923017a4007e9982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.019837 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d715a54286854b64bcf9535303460519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.023182 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7d583cf1f44fe79397e238d5d0a66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.024511 741 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7d583cf1f44fe79397e238d5d0a66e (DEFAULT_TABLE table=default.Table_d5bce1994c5a4329bb30179b053babbb [id=a437c9a8578b4ba6a9b20a5de4e35af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.024829 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7d583cf1f44fe79397e238d5d0a66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.025296 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c046302be54a4e91a532c737044d0f87" } modify_external_catalogs: true
I20250115 14:52:13.025485 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c046302be54a4e91a532c737044d0f87" } modify_external_catalogs: true
I20250115 14:52:12.996052 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 69a5e5c5758445d3b97615e82f91fcf7 (DEFAULT_TABLE table=default.Table_ee7134670e574f279ad4f5557f395e69 [id=4d5a034e3cd94f3fbb85cb8aea5d4527]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.030450 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a5e5c5758445d3b97615e82f91fcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.030736 756 tablet_service.cc:1467] Processing CreateTablet for tablet b4547b5be83e403db23a2ab92c7d0f6d (DEFAULT_TABLE table=default.Table_bc62cb62b8884088aa71a60746b07ffd [id=ad2bd57f7ddb42768c0f80fa87c516ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.031040 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4547b5be83e403db23a2ab92c7d0f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.032037 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2a1a69b6de5d433f82579b1259548fd2
I20250115 14:52:13.042670 1166 ts_tablet_manager.cc:1918] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:13.042807 1166 log.cc:1198] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/4f1ce59e05054a009cd36e12db35d449
I20250115 14:52:13.042824 755 tablet_service.cc:1467] Processing CreateTablet for tablet 19154be028d44ba2bed0f54ae04cc770 (DEFAULT_TABLE table=default.Table_7c4783895e7c4f9da20efe43779ce1e0 [id=fbfbae91fdde4a529a4e4b35f4699ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.043152 1166 ts_tablet_manager.cc:1939] T 4f1ce59e05054a009cd36e12db35d449 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.043277 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19154be028d44ba2bed0f54ae04cc770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.044030 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1a69b6de5d433f82579b1259548fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.044188 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 4f1ce59e05054a009cd36e12db35d449 (table default.Table_e209cc3424eb4349b5387a32ac96d6bc [id=18a209f910d44ef081b5925cd6ebe028]) successfully deleted
I20250115 14:52:13.044507 1166 tablet_replica.cc:331] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.044661 1166 raft_consensus.cc:2238] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.044770 1166 raft_consensus.cc:2267] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.051452 1167 ts_tablet_manager.cc:1918] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:13.052459 710 leader_election.cc:343] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 742792be08444792892bd9cd7336dda3
I20250115 14:52:13.052582 710 leader_election.cc:304] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:13.052791 2403 raft_consensus.cc:2743] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.054759 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1fc659e2d2d943a99aef530e7d71df88
I20250115 14:52:13.058475 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b44a481d86d48649c1a82a6b0ed9403. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.060714 1162 tablet_bootstrap.cc:492] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:13.060851 1162 ts_tablet_manager.cc:1397] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.197s user 0.004s sys 0.000s
I20250115 14:52:13.061134 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc659e2d2d943a99aef530e7d71df88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.066773 1166 ts_tablet_manager.cc:1905] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.075969 1164 tablet_bootstrap.cc:492] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.076110 1164 ts_tablet_manager.cc:1397] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.129s user 0.000s sys 0.003s
I20250115 14:52:13.076897 1164 raft_consensus.cc:357] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.077105 1164 raft_consensus.cc:383] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.077148 1164 raft_consensus.cc:738] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.077370 1164 consensus_queue.cc:260] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.081656 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1a69b6de5d433f82579b1259548fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.082559 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 32eee3d3dfec464baf10c238babb9eb1 (table default.Table_a14212e9310040fa88dd2432f80fdc58 [id=31fde8bd2aa3452fb638c948eb5d8a55]). Expected version 1 got 0
I20250115 14:52:13.083077 1164 ts_tablet_manager.cc:1428] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250115 14:52:13.058986 2341 raft_consensus.cc:491] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.083204 2341 raft_consensus.cc:513] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.083810 2341 leader_election.cc:290] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:13.084733 847 leader_election.cc:343] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: ba596cfb3ce64718a42a9083975edc8d
I20250115 14:52:13.071659 890 tablet_service.cc:1467] Processing CreateTablet for tablet b4547b5be83e403db23a2ab92c7d0f6d (DEFAULT_TABLE table=default.Table_bc62cb62b8884088aa71a60746b07ffd [id=ad2bd57f7ddb42768c0f80fa87c516ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.072248 878 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7d583cf1f44fe79397e238d5d0a66e (DEFAULT_TABLE table=default.Table_d5bce1994c5a4329bb30179b053babbb [id=a437c9a8578b4ba6a9b20a5de4e35af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.072630 884 tablet_service.cc:1467] Processing CreateTablet for tablet 69a5e5c5758445d3b97615e82f91fcf7 (DEFAULT_TABLE table=default.Table_ee7134670e574f279ad4f5557f395e69 [id=4d5a034e3cd94f3fbb85cb8aea5d4527]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.085177 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4547b5be83e403db23a2ab92c7d0f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.085347 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a5e5c5758445d3b97615e82f91fcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.086709 1162 raft_consensus.cc:357] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.086899 1162 raft_consensus.cc:383] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.086941 1162 raft_consensus.cc:738] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.087108 1162 consensus_queue.cc:260] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.087558 1162 ts_tablet_manager.cc:1428] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250115 14:52:13.072643 887 tablet_service.cc:1467] Processing CreateTablet for tablet d715a54286854b64bcf9535303460519 (DEFAULT_TABLE table=default.Table_9ab0ec2875a044198bb7906f36b7680b [id=6363c0b085dd4bf5923017a4007e9982]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.088529 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d715a54286854b64bcf9535303460519. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.088595 2563 raft_consensus.cc:491] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.088703 2563 raft_consensus.cc:513] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.089255 2563 leader_election.cc:290] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.089776 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:13.090199 983 leader_election.cc:343] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.090209 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:13.090732 984 leader_election.cc:343] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.090842 984 leader_election.cc:304] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.091156 2563 raft_consensus.cc:2743] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.091208 1164 tablet_bootstrap.cc:492] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.072876 889 tablet_service.cc:1467] Processing CreateTablet for tablet c487e956367845299e20cea05ce52987 (DEFAULT_TABLE table=default.Table_f4d5908220b548f1ba60227f435a948a [id=9e76d99bc062464580862b5b4bc023ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.092589 1164 tablet_bootstrap.cc:654] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.092614 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c487e956367845299e20cea05ce52987. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.093444 1167 log.cc:1198] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/1b2d04209fd34918b83978325a5e1df8
I20250115 14:52:13.093732 1167 ts_tablet_manager.cc:1939] T 1b2d04209fd34918b83978325a5e1df8 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.094125 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1080) took 1246 ms. Trace:
I20250115 14:52:13.094206 1167 rpcz_store.cc:276] 0115 14:52:11.847823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.847873 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:12.927404 (+1079531us) ts_tablet_manager.cc:1145] Deleting tablet 1b2d04209fd34918b83978325a5e1df8
0115 14:52:12.927413 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:12.966303 (+ 38890us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.051334 (+ 85031us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.094107 (+ 42773us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1079440}
I20250115 14:52:13.094506 1167 tablet_replica.cc:331] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.094748 1167 raft_consensus.cc:2238] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.094888 1167 raft_consensus.cc:2267] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.095372 1167 ts_tablet_manager.cc:1905] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.096398 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:13.073189 881 tablet_service.cc:1467] Processing CreateTablet for tablet 1b44a481d86d48649c1a82a6b0ed9403 (DEFAULT_TABLE table=default.Table_ca37f38ca7de45edba3a3cbcfae56bd4 [id=3c524adfd92349df9b4f74ab5c41a1cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.074088 876 tablet_service.cc:1467] Processing CreateTablet for tablet 2923dd60927d49f5b0ca09438cfd4ce3 (DEFAULT_TABLE table=default.Table_262d362a9eb447c689cbb66e877ab7be [id=7f7b3587d95c405ea2e217d74434180a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.096881 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b44a481d86d48649c1a82a6b0ed9403. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.097046 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2923dd60927d49f5b0ca09438cfd4ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.097944 1166 ts_tablet_manager.cc:1918] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.098043 1166 log.cc:1198] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b5937d6ad0494734a402fdc594eb9415
I20250115 14:52:13.098294 1166 ts_tablet_manager.cc:1939] T b5937d6ad0494734a402fdc594eb9415 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.098512 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 076278b53996407c8a5c73a389459eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.098836 874 tablet_service.cc:1514] Processing DeleteTablet for tablet a48227ec5aa846c3ac63bd29ed38be25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.099094 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc659e2d2d943a99aef530e7d71df88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.099843 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ca128bf3d6fc430298da932a78f54184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.099921 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba596cfb3ce64718a42a9083975edc8d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.074512 883 tablet_service.cc:1467] Processing CreateTablet for tablet 2955ecdd45364dc2934ba96a9a6c8e8b (DEFAULT_TABLE table=default.Table_1b5ca8c3048846588478dbfbebf1ff82 [id=8170a41205004f9ca83aa1c0ab7a2502]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.100530 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2955ecdd45364dc2934ba96a9a6c8e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.101909 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e616d78933e4f84b9ec2acc51450b18 with delete_type TABLET_DATA_DELETED (replaced by tablet ba33eb3f5a40450db5cf115fbe8a17e7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.102214 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 1b2d04209fd34918b83978325a5e1df8 (table default.Table_d548be616a2c4f01ac52eadd5c009eaf [id=1907295f0b5c49b9b99a188d4dc9f7f8]) successfully deleted
W20250115 14:52:13.102650 848 leader_election.cc:343] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.102753 848 leader_election.cc:304] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.074797 886 tablet_service.cc:1467] Processing CreateTablet for tablet a0d8d2e9c2c547debfb8ac410192a955 (DEFAULT_TABLE table=default.Table_4607c9a022124589b931dfbb075cefb5 [id=de749f3cbdc34b039fdb24c3927469d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.104543 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d8d2e9c2c547debfb8ac410192a955. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.104631 2341 raft_consensus.cc:2743] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.107498 1167 ts_tablet_manager.cc:1918] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.107581 1167 log.cc:1198] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/4f1ce59e05054a009cd36e12db35d449
I20250115 14:52:13.107527 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b5937d6ad0494734a402fdc594eb9415 (table default.Table_c2aefc9f5c604090bb4bd4f2326991ba [id=62ff1172217947669bf02f96134877c3]) successfully deleted
I20250115 14:52:13.107824 1166 tablet_replica.cc:331] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.107842 1167 ts_tablet_manager.cc:1939] T 4f1ce59e05054a009cd36e12db35d449 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.107995 1166 raft_consensus.cc:2238] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.108089 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1081) took 1260 ms. Trace:
I20250115 14:52:13.108201 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc659e2d2d943a99aef530e7d71df88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.075070 882 tablet_service.cc:1467] Processing CreateTablet for tablet 19154be028d44ba2bed0f54ae04cc770 (DEFAULT_TABLE table=default.Table_7c4783895e7c4f9da20efe43779ce1e0 [id=fbfbae91fdde4a529a4e4b35f4699ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.108153 1167 rpcz_store.cc:276] 0115 14:52:11.847907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.847976 (+ 69us) service_pool.cc:225] Handling call
0115 14:52:13.094462 (+1246486us) ts_tablet_manager.cc:1145] Deleting tablet 4f1ce59e05054a009cd36e12db35d449
0115 14:52:13.094471 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.104661 (+ 10190us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.107429 (+ 2768us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.108075 (+ 646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1246375}
I20250115 14:52:13.075238 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1a69b6de5d433f82579b1259548fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.108522 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1077) took 1260 ms. Trace:
I20250115 14:52:13.075359 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 005a4cd0e9f14e76b01afa49d946ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.075981 1168 tablet_bootstrap.cc:492] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.108600 1167 rpcz_store.cc:276] 0115 14:52:11.847559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.848060 (+ 501us) service_pool.cc:225] Handling call
0115 14:52:13.108433 (+1260373us) ts_tablet_manager.cc:1145] Deleting tablet 3f1e5d44b8064abbbf1b6419a1eca1ed
0115 14:52:13.108441 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.108507 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1260190}
I20250115 14:52:13.108791 1168 ts_tablet_manager.cc:1397] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.239s user 0.002s sys 0.001s
I20250115 14:52:13.108873 1167 tablet_replica.cc:331] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.108917 1166 raft_consensus.cc:2267] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.109057 1167 raft_consensus.cc:2238] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.109182 1167 raft_consensus.cc:2267] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.109758 1167 ts_tablet_manager.cc:1905] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.109656 1168 raft_consensus.cc:357] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.109896 1168 raft_consensus.cc:383] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.109949 1168 raft_consensus.cc:738] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.110038 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 4f1ce59e05054a009cd36e12db35d449 (table default.Table_e209cc3424eb4349b5387a32ac96d6bc [id=18a209f910d44ef081b5925cd6ebe028]) successfully deleted
I20250115 14:52:13.110196 1164 tablet_bootstrap.cc:492] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.110128 1168 consensus_queue.cc:260] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:13.110445 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3f1e5d44b8064abbbf1b6419a1eca1ed because the tablet was not found. No further retry: Not found: Tablet not found: 3f1e5d44b8064abbbf1b6419a1eca1ed
I20250115 14:52:13.110412 1164 ts_tablet_manager.cc:1397] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20250115 14:52:13.110576 1168 ts_tablet_manager.cc:1428] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:13.110862 1168 tablet_bootstrap.cc:492] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:13.081910 2549 consensus_queue.cc:1035] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:13.111845 1164 raft_consensus.cc:357] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.112141 1164 raft_consensus.cc:383] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.112254 1164 raft_consensus.cc:738] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.110445 1166 ts_tablet_manager.cc:1905] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.112628 1164 consensus_queue.cc:260] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.116029 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0610ab6883db4caa82a59fbf030bfedf" } modify_external_catalogs: true
I20250115 14:52:13.118510 1751 ts_tablet_manager.cc:1918] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.118603 1751 log.cc:1198] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/2e4c2707915e4e97ad6c66cb249c6d9e
I20250115 14:52:13.118883 1751 ts_tablet_manager.cc:1939] T 2e4c2707915e4e97ad6c66cb249c6d9e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:13.119807 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_221fbc5ac86b48eabef5ca465cce428d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.120414 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_221fbc5ac86b48eabef5ca465cce428d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.121351 2563 raft_consensus.cc:491] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.121474 2563 raft_consensus.cc:513] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.108538 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19154be028d44ba2bed0f54ae04cc770. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.124424 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b7d583cf1f44fe79397e238d5d0a66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.129822 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 32eee3d3dfec464baf10c238babb9eb1 (table default.Table_a14212e9310040fa88dd2432f80fdc58 [id=31fde8bd2aa3452fb638c948eb5d8a55]). Expected version 1 got 0
I20250115 14:52:13.131335 1166 ts_tablet_manager.cc:1918] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.131438 1166 log.cc:1198] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/742792be08444792892bd9cd7336dda3
I20250115 14:52:13.131728 1166 ts_tablet_manager.cc:1939] T 742792be08444792892bd9cd7336dda3 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.116220 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0610ab6883db4caa82a59fbf030bfedf" } modify_external_catalogs: true
I20250115 14:52:13.139389 755 tablet_service.cc:1467] Processing CreateTablet for tablet d36fa3146f0f40a59e5f96cce203fee0 (DEFAULT_TABLE table=default.Table_d47eeb687f8046f2803dd6b6a5e6c153 [id=96c32ebdf0834cbb845bc2cd4215ad88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.139796 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36fa3146f0f40a59e5f96cce203fee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.141953 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0985c835daab4ee9944a45c2af4b3f91" } modify_external_catalogs: true
I20250115 14:52:13.142160 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0985c835daab4ee9944a45c2af4b3f91" } modify_external_catalogs: true
I20250115 14:52:13.143352 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 742792be08444792892bd9cd7336dda3 (table default.Table_97c066067f7a49cab2dd5bdade9f6de2 [id=8c1c9324a0cb4ce5b5f64af989e4bdf6]) successfully deleted
I20250115 14:52:13.143412 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cac9e05c8cae4cab9a55c08b3dc25648" } modify_external_catalogs: true
I20250115 14:52:13.143550 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cac9e05c8cae4cab9a55c08b3dc25648" } modify_external_catalogs: true
I20250115 14:52:13.144096 2469 raft_consensus.cc:491] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.144225 2469 raft_consensus.cc:513] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.144841 2469 leader_election.cc:290] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.144757 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1d7ea0f078e148088d47b6e1b3b1903b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.145223 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d7ea0f078e148088d47b6e1b3b1903b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.145406 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b15de9bf8943cdb9e5abd943484993" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:13.145936 983 leader_election.cc:343] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.146452 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b15de9bf8943cdb9e5abd943484993" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:13.146926 984 leader_election.cc:343] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.147018 984 leader_election.cc:304] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.148166 2469 raft_consensus.cc:2743] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.148136 1011 tablet_service.cc:1467] Processing CreateTablet for tablet d36fa3146f0f40a59e5f96cce203fee0 (DEFAULT_TABLE table=default.Table_d47eeb687f8046f2803dd6b6a5e6c153 [id=96c32ebdf0834cbb845bc2cd4215ad88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.152882 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c
I20250115 14:52:13.153347 450 catalog_manager.cc:6596] ReplaceTablet: tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c deleted and replaced by tablet 27fc79ad43f14002a6fe1502a4347ea7
I20250115 14:52:13.153532 743 tablet_service.cc:1467] Processing CreateTablet for tablet dd6a942dfb1a4eb3a291521a99b1b2cd (DEFAULT_TABLE table=default.Table_10df3b88664a48a28d08136f0c29a7e2 [id=556e7ba7f3f745439a70d5705a8dabda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.153896 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6a942dfb1a4eb3a291521a99b1b2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.154690 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c with delete_type TABLET_DATA_DELETED (replaced by tablet 27fc79ad43f14002a6fe1502a4347ea7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.155229 1167 ts_tablet_manager.cc:1918] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.155355 1167 log.cc:1198] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b5937d6ad0494734a402fdc594eb9415
I20250115 14:52:13.155606 1167 ts_tablet_manager.cc:1939] T b5937d6ad0494734a402fdc594eb9415 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.155861 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1079) took 1308 ms. Trace:
I20250115 14:52:13.155927 1167 rpcz_store.cc:276] 0115 14:52:11.847724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.848276 (+ 552us) service_pool.cc:225] Handling call
0115 14:52:13.108832 (+1260556us) ts_tablet_manager.cc:1145] Deleting tablet b5937d6ad0494734a402fdc594eb9415
0115 14:52:13.108840 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.145517 (+ 36677us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.155132 (+ 9615us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.155846 (+ 714us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1260375}
I20250115 14:52:13.156260 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1082) took 1308 ms. Trace:
I20250115 14:52:13.156339 1167 rpcz_store.cc:276] 0115 14:52:11.848010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.848096 (+ 86us) service_pool.cc:225] Handling call
0115 14:52:13.156180 (+1308084us) ts_tablet_manager.cc:1145] Deleting tablet 338c0f6f39fd4335bbcda733b28b8bf8
0115 14:52:13.156188 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.156248 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1307629}
I20250115 14:52:13.156638 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1083) took 1282 ms. Trace:
I20250115 14:52:13.156706 1167 rpcz_store.cc:276] 0115 14:52:11.874276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.877496 (+ 3220us) service_pool.cc:225] Handling call
0115 14:52:13.156531 (+1279035us) ts_tablet_manager.cc:1145] Deleting tablet c6cb8dfa7c42480bbd139d5acf059a57
0115 14:52:13.156539 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.156621 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278868}
I20250115 14:52:13.156953 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1084) took 1275 ms. Trace:
I20250115 14:52:13.157019 1167 rpcz_store.cc:276] 0115 14:52:11.881852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.881904 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:13.156871 (+1274967us) ts_tablet_manager.cc:1145] Deleting tablet ad417f06081d41248dff6f1eed175f29
0115 14:52:13.156877 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.156940 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274600}
I20250115 14:52:13.157203 1162 tablet_bootstrap.cc:492] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:13.157281 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1085) took 1252 ms. Trace:
I20250115 14:52:13.157335 1167 rpcz_store.cc:276] 0115 14:52:11.904675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.904737 (+ 62us) service_pool.cc:225] Handling call
0115 14:52:13.157193 (+1252456us) ts_tablet_manager.cc:1145] Deleting tablet 3be094dcd3cb4f9d9259d354300c1600
0115 14:52:13.157201 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.157269 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1252259}
I20250115 14:52:13.157542 1167 tablet_replica.cc:331] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.157708 1167 raft_consensus.cc:2238] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.157842 1167 raft_consensus.cc:2267] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.158319 1167 ts_tablet_manager.cc:1905] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.159386 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f3626b709cb3478597936e4d350b38a1" } new_table_name: "default.Table_6a2e561fad40472588481d9c4b4f0f78" modify_external_catalogs: true
I20250115 14:52:13.159694 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c with delete_type TABLET_DATA_DELETED (replaced by tablet 27fc79ad43f14002a6fe1502a4347ea7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.159938 2563 leader_election.cc:290] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.160877 2341 raft_consensus.cc:491] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.160992 2341 raft_consensus.cc:513] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.161526 2341 leader_election.cc:290] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:13.162957 847 leader_election.cc:343] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: ca128bf3d6fc430298da932a78f54184
I20250115 14:52:13.163966 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4a8a9e000ade430aaa44bad5dd61df9f" } new_table_name: "default.Table_292c245cb6104cb0a2e350338cfff972" modify_external_catalogs: true
I20250115 14:52:13.164403 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cae4c9228c640e495507e8f7ad9f544" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:13.165182 983 leader_election.cc:343] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.166047 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1095) took 1170 ms. Trace:
I20250115 14:52:13.166126 1751 rpcz_store.cc:276] 0115 14:52:11.995330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.995386 (+ 56us) service_pool.cc:225] Handling call
0115 14:52:12.939570 (+944184us) ts_tablet_manager.cc:1145] Deleting tablet 2e4c2707915e4e97ad6c66cb249c6d9e
0115 14:52:12.939581 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.057129 (+117548us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.118399 (+ 61270us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.166028 (+ 47629us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":440,"tablet-delete.queue_time_us":944016}
I20250115 14:52:13.166476 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cae4c9228c640e495507e8f7ad9f544" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.167147 1751 tablet_replica.cc:331] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
W20250115 14:52:13.167179 984 leader_election.cc:343] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.167272 984 leader_election.cc:304] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.167321 1751 raft_consensus.cc:2238] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.167734 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36fa3146f0f40a59e5f96cce203fee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.168459 2469 raft_consensus.cc:2743] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.169100 1021 tablet_service.cc:1467] Processing CreateTablet for tablet dd6a942dfb1a4eb3a291521a99b1b2cd (DEFAULT_TABLE table=default.Table_10df3b88664a48a28d08136f0c29a7e2 [id=556e7ba7f3f745439a70d5705a8dabda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.169560 755 tablet_service.cc:1467] Processing CreateTablet for tablet 3b275886fee142aea689f420d1eb32b1 (DEFAULT_TABLE table=default.Table_28327356092a4fd1b277babda6f161dc [id=d700c963a6c24b5a88875e428da7b10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.169881 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b275886fee142aea689f420d1eb32b1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:13.170037 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0fe88a1365954ef5beb1d67b1cc4b747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe88a1365954ef5beb1d67b1cc4b747 already in progress: creating tablet
W20250115 14:52:13.170298 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 18f23815196047628d562da7312b010f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18f23815196047628d562da7312b010f already in progress: creating tablet
W20250115 14:52:13.170461 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 4e39e13fb8c84ae18bac5e45d576a0ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e39e13fb8c84ae18bac5e45d576a0ab already in progress: creating tablet
W20250115 14:52:13.170632 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 546fe492182f47b7b008f4719f97e877 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 546fe492182f47b7b008f4719f97e877 already in progress: creating tablet
W20250115 14:52:13.170819 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9022738b53c74ce99e8f35adb655d200 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9022738b53c74ce99e8f35adb655d200 already in progress: creating tablet
W20250115 14:52:13.171111 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 86c04fa23e7c40a5b5e07852e5d74a27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86c04fa23e7c40a5b5e07852e5d74a27 already in progress: creating tablet
I20250115 14:52:13.173159 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0c9709a31af8402eb6539db333df51a0" } new_table_name: "default.Table_93023df4f27c40058830c1e90a1b0243" modify_external_catalogs: true
I20250115 14:52:13.173795 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6a942dfb1a4eb3a291521a99b1b2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.174540 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_768b0287db3a4e8c9908083f55f84209"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.175014 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_768b0287db3a4e8c9908083f55f84209 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.165371 2563 consensus_queue.cc:1035] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250115 14:52:13.178673 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 00844e77811b4cbfb7240176055e2966 (DEFAULT_TABLE table=default.Table_aa12615b9ec74e788e7883960ac22d2a [id=ab12dc26bc004ca4b9a8a3aae448e9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.178839 1017 tablet_service.cc:1467] Processing CreateTablet for tablet c49eefbfc9ce404eb604fb265c6a64d0 (DEFAULT_TABLE table=default.Table_913e9383036c49f985fe9b1167c65efc [id=0f1cfe06c44c4195b4a11f462f2653b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.178874 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 30191e2c6e014b688d90ece93cd450fa (DEFAULT_TABLE table=default.Table_f4c2ee7746d24f588e97be9b31f6f62a [id=74a47455751f4ea6a42af05981aba3d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.179000 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00844e77811b4cbfb7240176055e2966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.179198 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30191e2c6e014b688d90ece93cd450fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.176151 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b134a43384d144c3a09a6a28068991ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.180881 1010 tablet_service.cc:1467] Processing CreateTablet for tablet d1512a0e99354e7096d4da71dcb95e02 (DEFAULT_TABLE table=default.Table_8db4c9b252ac4eaf90ab6b556fdbdefc [id=8c181283bbde4023851d60ec11a669c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.181181 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1512a0e99354e7096d4da71dcb95e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.182214 738 tablet_service.cc:1467] Processing CreateTablet for tablet 30191e2c6e014b688d90ece93cd450fa (DEFAULT_TABLE table=default.Table_f4c2ee7746d24f588e97be9b31f6f62a [id=74a47455751f4ea6a42af05981aba3d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.182561 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30191e2c6e014b688d90ece93cd450fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.182667 741 tablet_service.cc:1467] Processing CreateTablet for tablet c49eefbfc9ce404eb604fb265c6a64d0 (DEFAULT_TABLE table=default.Table_913e9383036c49f985fe9b1167c65efc [id=0f1cfe06c44c4195b4a11f462f2653b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.182942 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49eefbfc9ce404eb604fb265c6a64d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.184392 753 tablet_service.cc:1514] Processing DeleteTablet for tablet b5177265ba974ad6a381adb1e8c49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.184715 754 tablet_service.cc:1467] Processing CreateTablet for tablet 278f1fbe710043dc90571417e7e54390 (DEFAULT_TABLE table=default.Table_715e348f040b4062aeafb7c644b81772 [id=1ba8edc269c549ca8368f7a1059270ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.185002 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278f1fbe710043dc90571417e7e54390. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.185514 1166 tablet_replica.cc:331] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.185688 1166 raft_consensus.cc:2238] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.186244 750 tablet_service.cc:1467] Processing CreateTablet for tablet d1512a0e99354e7096d4da71dcb95e02 (DEFAULT_TABLE table=default.Table_8db4c9b252ac4eaf90ab6b556fdbdefc [id=8c181283bbde4023851d60ec11a669c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.186533 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1512a0e99354e7096d4da71dcb95e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.189831 1166 raft_consensus.cc:2267] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.190248 1168 tablet_bootstrap.cc:654] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.190721 756 tablet_service.cc:1467] Processing CreateTablet for tablet 00844e77811b4cbfb7240176055e2966 (DEFAULT_TABLE table=default.Table_aa12615b9ec74e788e7883960ac22d2a [id=ab12dc26bc004ca4b9a8a3aae448e9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.190979 1751 raft_consensus.cc:2267] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.191025 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00844e77811b4cbfb7240176055e2966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.192678 1166 ts_tablet_manager.cc:1905] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.195089 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49eefbfc9ce404eb604fb265c6a64d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.195935 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 3b275886fee142aea689f420d1eb32b1 (DEFAULT_TABLE table=default.Table_28327356092a4fd1b277babda6f161dc [id=d700c963a6c24b5a88875e428da7b10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.196341 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b275886fee142aea689f420d1eb32b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.199726 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 278f1fbe710043dc90571417e7e54390 (DEFAULT_TABLE table=default.Table_715e348f040b4062aeafb7c644b81772 [id=1ba8edc269c549ca8368f7a1059270ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.200053 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278f1fbe710043dc90571417e7e54390. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.200106 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6e6ffce5fa42f5a82f9b94982127e4 (DEFAULT_TABLE table=default.Table_454909c032fc4a9b82dd07b728e001ea [id=bb66ca33fd3b4396a49eb4a6b6f0848f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.200408 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6e6ffce5fa42f5a82f9b94982127e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.201457 1029 tablet_service.cc:1467] Processing CreateTablet for tablet e6821f4c34f042c8b4819176fe4418ff (DEFAULT_TABLE table=default.Table_19d4a8e242364fe4937cdbd66c57029d [id=ac030a4c81a444b399fedcdc1a76a47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.201742 1751 ts_tablet_manager.cc:1905] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.201778 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6821f4c34f042c8b4819176fe4418ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.201985 2403 raft_consensus.cc:491] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.199810 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdebf6cf072452ea778009767e009c8 (DEFAULT_TABLE table=default.Table_db34dca850e84ecc810cb875e2b237cd [id=2e6f13c402c54af8be105fad905c3123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.202113 2403 raft_consensus.cc:513] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.202733 2403 leader_election.cc:290] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.202847 2403 raft_consensus.cc:491] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.202935 2403 raft_consensus.cc:513] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:13.180866 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b134a43384d144c3a09a6a28068991ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.183557 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2a005ad11e7941fab96b5565d55cfaf7
W20250115 14:52:13.183125 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a300a76b8f7942dea648a20cbb3d5368 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a300a76b8f7942dea648a20cbb3d5368 already in progress: creating tablet
I20250115 14:52:13.203387 2403 leader_election.cc:290] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:13.203855 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d71a1ecd900942e5ae788c9a28c70a7a (table default.Table_0c9709a31af8402eb6539db333df51a0 [id=eec67f558ecb4cffaa82516d5e51906d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d71a1ecd900942e5ae788c9a28c70a7a consensus state has no leader
I20250115 14:52:13.204002 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fff787810e9e4b10aa7c90a8f936a7e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.204420 753 tablet_service.cc:1467] Processing CreateTablet for tablet b5a476872b254f6989640209077bf38c (DEFAULT_TABLE table=default.Table_19880e28f59446b0a4b4a8cc082428b7 [id=28e483eaf07c438d97a979706cb3195b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:13.204618 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fff787810e9e4b10aa7c90a8f936a7e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.204774 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5a476872b254f6989640209077bf38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.204684 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eee3d3dfec464baf10c238babb9eb1" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:13.204857 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32eee3d3dfec464baf10c238babb9eb1" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:13.205372 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c34919648447fac14208ede0ddc01" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:13.205565 1048 raft_consensus.cc:2388] T ec8c34919648447fac14208ede0ddc01 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d7ecece2ab49e1be7079216517d13b in current term 1: Already voted for candidate 73c86158cc2641f182ea50a06634592d in this term.
I20250115 14:52:13.180262 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8ea6902fe95d4b84bdaf21fd1a87e38a" } new_table_name: "default.Table_6a2171304c82499ea4dd40cf34994b06" modify_external_catalogs: true
I20250115 14:52:13.206032 755 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6e6ffce5fa42f5a82f9b94982127e4 (DEFAULT_TABLE table=default.Table_454909c032fc4a9b82dd07b728e001ea [id=bb66ca33fd3b4396a49eb4a6b6f0848f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.206332 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6e6ffce5fa42f5a82f9b94982127e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.207234 1166 ts_tablet_manager.cc:1918] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.207331 1166 log.cc:1198] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/97b599c6b8174959b8b66e90e9242057
I20250115 14:52:13.207604 1166 ts_tablet_manager.cc:1939] T 97b599c6b8174959b8b66e90e9242057 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.208398 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a005ad11e7941fab96b5565d55cfaf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.208668 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c34919648447fac14208ede0ddc01" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:13.209124 776 raft_consensus.cc:3054] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:13.209939 741 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdebf6cf072452ea778009767e009c8 (DEFAULT_TABLE table=default.Table_db34dca850e84ecc810cb875e2b237cd [id=2e6f13c402c54af8be105fad905c3123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.210232 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cdebf6cf072452ea778009767e009c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.213198 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cdebf6cf072452ea778009767e009c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.214020 710 leader_election.cc:304] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:13.214388 710 leader_election.cc:304] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:13.214713 2403 raft_consensus.cc:2743] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.214871 2403 raft_consensus.cc:2743] T ec8c34919648447fac14208ede0ddc01 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.217767 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 97b599c6b8174959b8b66e90e9242057 (table default.Table_e3e5361d9cb144999d02ff4df3e1999b [id=5063a3c3ef3a4b1794a2c8532ab7adf6]) successfully deleted
W20250115 14:52:13.218319 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ac97cd8e11cf4b5e8978902d1c99aad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac97cd8e11cf4b5e8978902d1c99aad8 already in progress: creating tablet
I20250115 14:52:13.219880 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca128bf3d6fc430298da932a78f54184" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:13.219990 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2a1a69b6de5d433f82579b1259548fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1a69b6de5d433f82579b1259548fd2 already in progress: creating tablet
W20250115 14:52:13.220487 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet cc9701156b0e48498666e34b3bec3043 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9701156b0e48498666e34b3bec3043 already in progress: creating tablet
W20250115 14:52:13.220927 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1fc659e2d2d943a99aef530e7d71df88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fc659e2d2d943a99aef530e7d71df88 already in progress: creating tablet
W20250115 14:52:13.221498 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c already in progress: creating tablet
W20250115 14:52:13.223403 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b5177265ba974ad6a381adb1e8c49589 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5177265ba974ad6a381adb1e8c49589 already in progress: creating tablet
W20250115 14:52:13.223848 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2a005ad11e7941fab96b5565d55cfaf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a005ad11e7941fab96b5565d55cfaf7 already in progress: creating tablet
I20250115 14:52:13.227643 1162 tablet_bootstrap.cc:654] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.228137 2563 raft_consensus.cc:491] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.228267 2563 raft_consensus.cc:513] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.228922 2563 leader_election.cc:290] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.229638 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd94e4a7fef44f5be462e7a77e8a123" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:13.230401 983 leader_election.cc:343] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.231168 882 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fe51b51994491ab38a644bfc55e3ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.234571 882 tablet_service.cc:1467] Processing CreateTablet for tablet dd6a942dfb1a4eb3a291521a99b1b2cd (DEFAULT_TABLE table=default.Table_10df3b88664a48a28d08136f0c29a7e2 [id=556e7ba7f3f745439a70d5705a8dabda]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.235186 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6a942dfb1a4eb3a291521a99b1b2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.241043 2549 raft_consensus.cc:491] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.241187 2549 raft_consensus.cc:513] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.241789 2549 leader_election.cc:290] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:13.243633 847 leader_election.cc:343] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 076278b53996407c8a5c73a389459eaa
I20250115 14:52:13.246212 1167 ts_tablet_manager.cc:1918] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.246316 1167 log.cc:1198] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/0651be54e0b244a4bace9f47efc9b58a
I20250115 14:52:13.246594 1167 ts_tablet_manager.cc:1939] T 0651be54e0b244a4bace9f47efc9b58a P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.248189 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a005ad11e7941fab96b5565d55cfaf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.248627 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b5937d6ad0494734a402fdc594eb9415 (table default.Table_c2aefc9f5c604090bb4bd4f2326991ba [id=62ff1172217947669bf02f96134877c3]) successfully deleted
I20250115 14:52:13.248795 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1086) took 1280 ms. Trace:
W20250115 14:52:13.248931 440 catalog_manager.cc:4839] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 338c0f6f39fd4335bbcda733b28b8bf8 because the tablet was not found. No further retry: Not found: Tablet not found: 338c0f6f39fd4335bbcda733b28b8bf8
W20250115 14:52:13.249146 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c6cb8dfa7c42480bbd139d5acf059a57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6cb8dfa7c42480bbd139d5acf059a57 already in progress: creating tablet
I20250115 14:52:13.248935 1167 rpcz_store.cc:276] 0115 14:52:11.966393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:11.966441 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:13.157505 (+1191064us) ts_tablet_manager.cc:1145] Deleting tablet 0651be54e0b244a4bace9f47efc9b58a
0115 14:52:13.157513 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.229110 (+ 71597us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.246103 (+ 16993us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.246877 (+ 774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1190898}
W20250115 14:52:13.249358 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ad417f06081d41248dff6f1eed175f29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad417f06081d41248dff6f1eed175f29 already in progress: creating tablet
W20250115 14:52:13.249536 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3be094dcd3cb4f9d9259d354300c1600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3be094dcd3cb4f9d9259d354300c1600 already in progress: creating tablet
I20250115 14:52:13.254690 1024 tablet_service.cc:1467] Processing CreateTablet for tablet b5a476872b254f6989640209077bf38c (DEFAULT_TABLE table=default.Table_19880e28f59446b0a4b4a8cc082428b7 [id=28e483eaf07c438d97a979706cb3195b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.255110 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5a476872b254f6989640209077bf38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.252952 2549 raft_consensus.cc:491] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.255288 2549 raft_consensus.cc:513] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.254848 2655 tablet.cc:1721] T 32eee3d3dfec464baf10c238babb9eb1 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:13.255919 2549 leader_election.cc:290] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:13.257079 847 leader_election.cc:343] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 005a4cd0e9f14e76b01afa49d946ee32
I20250115 14:52:13.257460 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1093) took 1015 ms. Trace:
I20250115 14:52:13.260636 1167 rpcz_store.cc:276] 0115 14:52:12.242063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:12.242231 (+ 168us) service_pool.cc:225] Handling call
0115 14:52:13.257344 (+1015113us) ts_tablet_manager.cc:1145] Deleting tablet 0fe88a1365954ef5beb1d67b1cc4b747
0115 14:52:13.257355 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.257444 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1014952}
I20250115 14:52:13.261529 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d5ffa997ca6c46559830b1dedf2d1a66
I20250115 14:52:13.262233 741 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ffa997ca6c46559830b1dedf2d1a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.262610 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1096) took 1016 ms. Trace:
I20250115 14:52:13.262681 1167 rpcz_store.cc:276] 0115 14:52:12.246183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:12.246242 (+ 59us) service_pool.cc:225] Handling call
0115 14:52:13.262497 (+1016255us) ts_tablet_manager.cc:1145] Deleting tablet 18f23815196047628d562da7312b010f
0115 14:52:13.262508 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.262595 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1016066}
I20250115 14:52:13.262929 1167 tablet_replica.cc:331] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.263121 1167 raft_consensus.cc:2238] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.263233 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ffa997ca6c46559830b1dedf2d1a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.263273 1167 raft_consensus.cc:2267] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.263518 1166 tablet_replica.cc:331] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.263701 1166 raft_consensus.cc:2238] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.263780 1167 ts_tablet_manager.cc:1905] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.263887 1166 raft_consensus.cc:2267] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.264022 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 32eee3d3dfec464baf10c238babb9eb1 (table default.Table_a14212e9310040fa88dd2432f80fdc58 [id=31fde8bd2aa3452fb638c948eb5d8a55]). Expected version 1 got 0
I20250115 14:52:13.264992 1166 ts_tablet_manager.cc:1905] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.264864 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c586d1fdfee24db2b1b3e9196fca1641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.265415 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c586d1fdfee24db2b1b3e9196fca1641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.266548 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0936fa510a454b8e91d6ef2b345b18df
I20250115 14:52:13.267241 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dd78a3cf930047c2b4181c0498fa2f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.267428 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 0936fa510a454b8e91d6ef2b345b18df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:13.267697 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd78a3cf930047c2b4181c0498fa2f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.267853 1018 tablet_service.cc:1514] Processing DeleteTablet for tablet 0936fa510a454b8e91d6ef2b345b18df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.270442 738 tablet_service.cc:1467] Processing CreateTablet for tablet e6821f4c34f042c8b4819176fe4418ff (DEFAULT_TABLE table=default.Table_19d4a8e242364fe4937cdbd66c57029d [id=ac030a4c81a444b399fedcdc1a76a47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.270789 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6821f4c34f042c8b4819176fe4418ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:13.272071 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b252e48108947cbbb36335499b51c92 consensus state has no leader
W20250115 14:52:13.272270 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9fe7404aa6ed49ca83d53d5d6e7d532c consensus state has no leader
I20250115 14:52:13.272651 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e9629998fb1e463cad667d917dd31dee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.273160 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9629998fb1e463cad667d917dd31dee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.273180 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_dd3cdaff78cf4f24b0ea6ba66ecf8941"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.273618 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd3cdaff78cf4f24b0ea6ba66ecf8941 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:13.274398 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99ae2f8bd8fe4e92995dddf414172f0f consensus state has no leader
I20250115 14:52:13.274744 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0be69f26c9784abb83eee94fe5bfdc39" } new_table_name: "default.Table_226f0457e628445a92b609db89b91417" modify_external_catalogs: true
W20250115 14:52:13.275964 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 324c810773f249db9e8325b09805141a consensus state has no leader
I20250115 14:52:13.276355 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_134a6fa6193243e69b7febda51198916"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.276762 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_134a6fa6193243e69b7febda51198916 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.278064 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f8f4fd3823584b899089dc4b570b2756" } modify_external_catalogs: true
I20250115 14:52:13.278259 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f8f4fd3823584b899089dc4b570b2756" } modify_external_catalogs: true
I20250115 14:52:13.280723 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_18573a0a9c86499d9223ac71aa866a24" } modify_external_catalogs: true
I20250115 14:52:13.280889 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_18573a0a9c86499d9223ac71aa866a24" } modify_external_catalogs: true
I20250115 14:52:13.286329 1751 ts_tablet_manager.cc:1918] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.286445 1751 log.cc:1198] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/8f8bae9867d745afb1dfdf47bcb881da
I20250115 14:52:13.286758 1751 ts_tablet_manager.cc:1939] T 8f8bae9867d745afb1dfdf47bcb881da P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:13.289039 1167 ts_tablet_manager.cc:1918] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.289253 1167 log.cc:1198] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b803d8e103a5423f85c75c9006809b17
I20250115 14:52:13.289606 1167 ts_tablet_manager.cc:1939] T b803d8e103a5423f85c75c9006809b17 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.290091 1167 tablet_replica.cc:331] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.290359 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c with delete_type TABLET_DATA_DELETED (replaced by tablet 27fc79ad43f14002a6fe1502a4347ea7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.294062 876 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6e6ffce5fa42f5a82f9b94982127e4 (DEFAULT_TABLE table=default.Table_454909c032fc4a9b82dd07b728e001ea [id=bb66ca33fd3b4396a49eb4a6b6f0848f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.290809 887 tablet_service.cc:1467] Processing CreateTablet for tablet 3b275886fee142aea689f420d1eb32b1 (DEFAULT_TABLE table=default.Table_28327356092a4fd1b277babda6f161dc [id=d700c963a6c24b5a88875e428da7b10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.294430 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6e6ffce5fa42f5a82f9b94982127e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.294615 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b275886fee142aea689f420d1eb32b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.298450 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6308f68e41554470b7d1321c60d4eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.291275 883 tablet_service.cc:1467] Processing CreateTablet for tablet 00844e77811b4cbfb7240176055e2966 (DEFAULT_TABLE table=default.Table_aa12615b9ec74e788e7883960ac22d2a [id=ab12dc26bc004ca4b9a8a3aae448e9ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.300390 1167 raft_consensus.cc:2238] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.300493 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00844e77811b4cbfb7240176055e2966. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.300557 1167 raft_consensus.cc:2267] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.301090 1167 ts_tablet_manager.cc:1905] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.291672 886 tablet_service.cc:1467] Processing CreateTablet for tablet c49eefbfc9ce404eb604fb265c6a64d0 (DEFAULT_TABLE table=default.Table_913e9383036c49f985fe9b1167c65efc [id=0f1cfe06c44c4195b4a11f462f2653b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.304576 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c49eefbfc9ce404eb604fb265c6a64d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.307893 1166 ts_tablet_manager.cc:1918] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:13.307997 1166 log.cc:1198] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d5ffa997ca6c46559830b1dedf2d1a66
I20250115 14:52:13.292068 881 tablet_service.cc:1467] Processing CreateTablet for tablet d1512a0e99354e7096d4da71dcb95e02 (DEFAULT_TABLE table=default.Table_8db4c9b252ac4eaf90ab6b556fdbdefc [id=8c181283bbde4023851d60ec11a669c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.308799 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1512a0e99354e7096d4da71dcb95e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.311043 1166 ts_tablet_manager.cc:1939] T d5ffa997ca6c46559830b1dedf2d1a66 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.292479 884 tablet_service.cc:1467] Processing CreateTablet for tablet 30191e2c6e014b688d90ece93cd450fa (DEFAULT_TABLE table=default.Table_f4c2ee7746d24f588e97be9b31f6f62a [id=74a47455751f4ea6a42af05981aba3d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.312759 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30191e2c6e014b688d90ece93cd450fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.313949 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]) successfully deleted
W20250115 14:52:13.314837 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0936fa510a454b8e91d6ef2b345b18df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0936fa510a454b8e91d6ef2b345b18df already in progress: creating tablet
I20250115 14:52:13.315685 2656 tablet.cc:1721] T 32eee3d3dfec464baf10c238babb9eb1 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:13.292912 889 tablet_service.cc:1467] Processing CreateTablet for tablet 278f1fbe710043dc90571417e7e54390 (DEFAULT_TABLE table=default.Table_715e348f040b4062aeafb7c644b81772 [id=1ba8edc269c549ca8368f7a1059270ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.293296 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdebf6cf072452ea778009767e009c8 (DEFAULT_TABLE table=default.Table_db34dca850e84ecc810cb875e2b237cd [id=2e6f13c402c54af8be105fad905c3123]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.293675 891 tablet_service.cc:1467] Processing CreateTablet for tablet e6821f4c34f042c8b4819176fe4418ff (DEFAULT_TABLE table=default.Table_19d4a8e242364fe4937cdbd66c57029d [id=ac030a4c81a444b399fedcdc1a76a47e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.317106 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278f1fbe710043dc90571417e7e54390. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.318492 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6821f4c34f042c8b4819176fe4418ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.320067 1162 tablet_bootstrap.cc:492] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:13.320190 1162 ts_tablet_manager.cc:1397] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.163s user 0.013s sys 0.003s
I20250115 14:52:13.320880 1162 raft_consensus.cc:357] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.321054 1162 raft_consensus.cc:383] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.321069 1167 ts_tablet_manager.cc:1918] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.321102 1162 raft_consensus.cc:738] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.321195 1167 log.cc:1198] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/580742eb5dac4dba9995809890e0e315
I20250115 14:52:13.321466 1167 ts_tablet_manager.cc:1939] T 580742eb5dac4dba9995809890e0e315 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.323190 2563 consensus_queue.cc:1035] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:13.323335 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cdebf6cf072452ea778009767e009c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.331841 1167 tablet_replica.cc:331] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.332165 1167 raft_consensus.cc:2238] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:13.332554 1167 raft_consensus.cc:2267] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.336864 1167 ts_tablet_manager.cc:1905] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.344341 882 tablet_service.cc:1467] Processing CreateTablet for tablet d36fa3146f0f40a59e5f96cce203fee0 (DEFAULT_TABLE table=default.Table_d47eeb687f8046f2803dd6b6a5e6c153 [id=96c32ebdf0834cbb845bc2cd4215ad88]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.344738 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d36fa3146f0f40a59e5f96cce203fee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.344786 1751 tablet_replica.cc:331] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:13.345028 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 8f8bae9867d745afb1dfdf47bcb881da (table default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]) successfully deleted
I20250115 14:52:13.346139 1164 ts_tablet_manager.cc:1428] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.236s user 0.002s sys 0.000s
I20250115 14:52:13.348029 1164 tablet_bootstrap.cc:492] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.349697 1164 tablet_bootstrap.cc:654] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.352455 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 32eee3d3dfec464baf10c238babb9eb1 (table default.Table_a14212e9310040fa88dd2432f80fdc58 [id=31fde8bd2aa3452fb638c948eb5d8a55]). Expected version 1 got 0
I20250115 14:52:13.342423 1162 consensus_queue.cc:260] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.355146 1162 ts_tablet_manager.cc:1428] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.035s user 0.000s sys 0.002s
I20250115 14:52:13.356221 1162 tablet_bootstrap.cc:492] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:13.362249 1167 ts_tablet_manager.cc:1918] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:13.362365 1167 log.cc:1198] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/7233ebf3bae1462d8eeb3669b39944c0
I20250115 14:52:13.362694 1167 ts_tablet_manager.cc:1939] T 7233ebf3bae1462d8eeb3669b39944c0 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.363400 1167 tablet_replica.cc:331] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.363580 1167 raft_consensus.cc:2238] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.363718 1167 raft_consensus.cc:2267] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.364250 1167 ts_tablet_manager.cc:1905] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.378453 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c02ac9ab9ef347c0ab6f30ececeb28a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.379207 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c02ac9ab9ef347c0ab6f30ececeb28a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.381721 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 0936fa510a454b8e91d6ef2b345b18df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.381975 883 tablet_service.cc:1514] Processing DeleteTablet for tablet d5ffa997ca6c46559830b1dedf2d1a66 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.382198 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a005ad11e7941fab96b5565d55cfaf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.382562 1162 tablet_bootstrap.cc:654] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.382577 884 tablet_service.cc:1467] Processing CreateTablet for tablet b5a476872b254f6989640209077bf38c (DEFAULT_TABLE table=default.Table_19880e28f59446b0a4b4a8cc082428b7 [id=28e483eaf07c438d97a979706cb3195b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.382694 2341 consensus_queue.cc:1035] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:13.383090 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5a476872b254f6989640209077bf38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.390053 2657 consensus_queue.cc:1035] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:13.393042 1168 tablet_bootstrap.cc:492] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.393182 1168 ts_tablet_manager.cc:1397] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.282s user 0.006s sys 0.010s
I20250115 14:52:13.393925 1168 raft_consensus.cc:357] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.394102 1168 raft_consensus.cc:383] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.394408 1168 raft_consensus.cc:738] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.394667 1168 consensus_queue.cc:260] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.394973 2655 tablet.cc:1721] T ec2a8000f5474f62aae52c8e88e45276 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:13.395884 1168 ts_tablet_manager.cc:1428] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:13.399674 1168 tablet_bootstrap.cc:492] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:13.401258 2549 consensus_queue.cc:1035] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:13.404404 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 0651be54e0b244a4bace9f47efc9b58a (table default.Table_93023df4f27c40058830c1e90a1b0243 [id=eec67f558ecb4cffaa82516d5e51906d]) successfully deleted
W20250115 14:52:13.405798 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0fe88a1365954ef5beb1d67b1cc4b747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0fe88a1365954ef5beb1d67b1cc4b747 already in progress: creating tablet
W20250115 14:52:13.406273 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 18f23815196047628d562da7312b010f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18f23815196047628d562da7312b010f already in progress: creating tablet
I20250115 14:52:13.406683 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b803d8e103a5423f85c75c9006809b17 (table default.Table_4dff2e869f8141e09dd757d6e47118f4 [id=43852683a5a64e1b85403395fa3d210c]) successfully deleted
I20250115 14:52:13.407090 1167 ts_tablet_manager.cc:1918] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.407122 738 tablet_service.cc:1514] Processing DeleteTablet for tablet b5177265ba974ad6a381adb1e8c49589 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.407364 1166 tablet_replica.cc:331] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.407558 1166 raft_consensus.cc:2238] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.407706 1166 raft_consensus.cc:2267] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.408831 1166 ts_tablet_manager.cc:1905] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.409087 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 580742eb5dac4dba9995809890e0e315 (table default.Table_d8d3e6a881ce475a8f1319238ab8c610 [id=b6de0dbbced44c83af5f98a9d1020cfc]) successfully deleted
W20250115 14:52:13.409540 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9022738b53c74ce99e8f35adb655d200 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9022738b53c74ce99e8f35adb655d200 already in progress: creating tablet
W20250115 14:52:13.410229 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 86c04fa23e7c40a5b5e07852e5d74a27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86c04fa23e7c40a5b5e07852e5d74a27 already in progress: creating tablet
I20250115 14:52:13.410320 1164 tablet_bootstrap.cc:492] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.410511 1164 ts_tablet_manager.cc:1397] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20250115 14:52:13.411244 1164 raft_consensus.cc:357] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:13.411815 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 546fe492182f47b7b008f4719f97e877 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 546fe492182f47b7b008f4719f97e877 already in progress: creating tablet
W20250115 14:52:13.412089 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 4e39e13fb8c84ae18bac5e45d576a0ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e39e13fb8c84ae18bac5e45d576a0ab already in progress: creating tablet
I20250115 14:52:13.413448 1162 tablet_bootstrap.cc:492] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:13.413623 1162 ts_tablet_manager.cc:1397] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.002s
I20250115 14:52:13.412564 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd94e4a7fef44f5be462e7a77e8a123" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.413825 1164 raft_consensus.cc:383] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.414021 1164 raft_consensus.cc:738] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.414335 1164 consensus_queue.cc:260] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.415894 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_1b5ca8c3048846588478dbfbebf1ff82 [id=8170a41205004f9ca83aa1c0ab7a2502]) successfully deleted
I20250115 14:52:13.414850 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076278b53996407c8a5c73a389459eaa" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:13.416195 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 6afbdd0b585e4e7e82067dce061924d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6afbdd0b585e4e7e82067dce061924d0 already in progress: creating tablet
I20250115 14:52:13.415192 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005a4cd0e9f14e76b01afa49d946ee32" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:13.416447 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 856533ed254a49b4a461ed1ad3eb405e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 856533ed254a49b4a461ed1ad3eb405e already in progress: creating tablet
I20250115 14:52:13.416476 1164 ts_tablet_manager.cc:1428] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250115 14:52:13.416678 1167 log.cc:1198] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/4294c49da1e044be993100b3cf5e2cea
I20250115 14:52:13.417155 1167 ts_tablet_manager.cc:1939] T 4294c49da1e044be993100b3cf5e2cea P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.417344 2656 tablet.cc:1721] T ec2a8000f5474f62aae52c8e88e45276 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:13.421586 1164 tablet_bootstrap.cc:492] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.422271 1167 tablet_replica.cc:331] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.413434 2660 tablet.cc:1721] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:52:13.424957 984 leader_election.cc:343] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.425092 984 leader_election.cc:304] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.425287 2564 raft_consensus.cc:2743] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:13.427039 984 consensus_peers.cc:487] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97b599c6b8174959b8b66e90e9242057. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:13.427358 1167 raft_consensus.cc:2238] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.428820 1167 raft_consensus.cc:2267] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.430176 1167 ts_tablet_manager.cc:1905] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.433806 1162 raft_consensus.cc:357] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.436851 1164 tablet_bootstrap.cc:654] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.442461 451 master_service.cc:946] ReplaceTablet: received request to replace tablet 0f82d944a7e04ace8e4addb3897d2c8e from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:13.443131 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_19d4a8e242364fe4937cdbd66c57029d" } modify_external_catalogs: true
I20250115 14:52:13.443332 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_19d4a8e242364fe4937cdbd66c57029d" } modify_external_catalogs: true
I20250115 14:52:13.443804 1168 tablet_bootstrap.cc:654] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.442760 738 tablet_service.cc:1514] Processing DeleteTablet for tablet ca128bf3d6fc430298da932a78f54184 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:10 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:13.444969 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d40be9c7d719498896a93c07a549280c consensus state has no leader
I20250115 14:52:13.445609 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c497fb29752d4ed7bda207301f90cf05
I20250115 14:52:13.441282 1751 raft_consensus.cc:2238] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.445832 1751 raft_consensus.cc:2267] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.446389 1751 ts_tablet_manager.cc:1905] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.447058 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2c89d5dd5dd74df2a66e50636cc5e7f9
I20250115 14:52:13.453430 1751 ts_tablet_manager.cc:1918] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.453542 1751 log.cc:1198] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ded663ac1197454bbde3c4c2d526002d
I20250115 14:52:13.453819 1751 ts_tablet_manager.cc:1939] T ded663ac1197454bbde3c4c2d526002d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:13.454567 738 tablet_service.cc:1514] Processing DeleteTablet for tablet c497fb29752d4ed7bda207301f90cf05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.454762 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.454916 1751 tablet_replica.cc:331] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:13.455086 1751 raft_consensus.cc:2238] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.455204 1751 raft_consensus.cc:2267] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.455637 1751 ts_tablet_manager.cc:1905] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.458782 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_a14212e9310040fa88dd2432f80fdc58 [id=31fde8bd2aa3452fb638c948eb5d8a55] alter complete (version 1)
I20250115 14:52:13.459623 1162 raft_consensus.cc:383] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.459784 1162 raft_consensus.cc:738] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.460108 1167 ts_tablet_manager.cc:1918] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.460229 1162 consensus_queue.cc:260] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.461138 1162 ts_tablet_manager.cc:1428] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.047s user 0.001s sys 0.000s
I20250115 14:52:13.462023 1164 tablet_bootstrap.cc:492] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.462216 1164 ts_tablet_manager.cc:1397] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20250115 14:52:13.463063 1164 raft_consensus.cc:357] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.463313 1164 raft_consensus.cc:383] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.463416 1164 raft_consensus.cc:738] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.463680 1164 consensus_queue.cc:260] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.463850 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e6821f4c34f042c8b4819176fe4418ff
I20250115 14:52:13.464820 743 tablet_service.cc:1514] Processing DeleteTablet for tablet e6821f4c34f042c8b4819176fe4418ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.466084 1164 ts_tablet_manager.cc:1428] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250115 14:52:13.466452 450 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f] alter complete (version 1)
I20250115 14:52:13.466557 1164 tablet_bootstrap.cc:492] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.467005 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8a376696a65648979c5f5791bc936fab" } modify_external_catalogs: true
I20250115 14:52:13.467248 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8a376696a65648979c5f5791bc936fab" } modify_external_catalogs: true
I20250115 14:52:13.468115 1164 tablet_bootstrap.cc:654] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.468195 1162 tablet_bootstrap.cc:492] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:13.470363 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8a480bbe40e04cbcbcd51a7e214bfd42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.470865 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a480bbe40e04cbcbcd51a7e214bfd42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.474306 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5ce3a92ee6154dee8af40b3ac625550c" } new_table_name: "default.Table_60224e46d88a4289a7e81d10e657c0cd" modify_external_catalogs: true
I20250115 14:52:13.475701 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_39f9c4e24e5a4e54bde4760f73b4ff65" } new_table_name: "default.Table_ffef6bc5c1df419aac950ae0bb25cff7" modify_external_catalogs: true
I20250115 14:52:13.474601 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 580742eb5dac4dba9995809890e0e315 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.477186 1167 log.cc:1198] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/3b82924e735846b2914520684d1b8fc7
I20250115 14:52:13.478168 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0f7932fd4ac249c18e0f79b0acb45cc2" } new_table_name: "default.Table_162eb94aa1674d21ae644021401a54e8" modify_external_catalogs: true
I20250115 14:52:13.479363 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c3d7415d4479485b9629973554670e05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.479846 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3d7415d4479485b9629973554670e05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.482415 1167 ts_tablet_manager.cc:1939] T 3b82924e735846b2914520684d1b8fc7 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.482841 1167 tablet_replica.cc:331] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.484309 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5fc99dfa3f504b2f9a09dcfc542dde73" } modify_external_catalogs: true
I20250115 14:52:13.484522 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5fc99dfa3f504b2f9a09dcfc542dde73" } modify_external_catalogs: true
I20250115 14:52:13.485216 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_044eb9404b5446298ecd6e8f48215b59" } new_table_name: "default.Table_d01041829a3b4c529cf198b45e9a0d3f" modify_external_catalogs: true
I20250115 14:52:13.485385 1751 ts_tablet_manager.cc:1918] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.485496 1751 log.cc:1198] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a48a0af9422f43cf86531665c0fa1aee
I20250115 14:52:13.485786 1751 ts_tablet_manager.cc:1939] T a48a0af9422f43cf86531665c0fa1aee P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:13.487658 1167 raft_consensus.cc:2238] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.487913 1167 raft_consensus.cc:2267] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.488524 1167 ts_tablet_manager.cc:1905] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:13.489558 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ad988e25800f4af9a069b5a80582f749 consensus state has no leader
I20250115 14:52:13.490049 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7c016ac18c7c4026b8e60a53f889bf0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.490545 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c016ac18c7c4026b8e60a53f889bf0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.492098 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7c37ffc6335846248f463334008c0fe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.493352 1162 tablet_bootstrap.cc:654] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:13.494053 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c37ffc6335846248f463334008c0fe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:13.492822 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c3d882a2515d4f8aa7f4fd7aa776f4e4 consensus state has no leader
W20250115 14:52:13.493072 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9107b85c08c646cab34f2128ea8db84f consensus state has no leader
I20250115 14:52:13.490715 1168 tablet_bootstrap.cc:492] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.496095 1168 ts_tablet_manager.cc:1397] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.000s
I20250115 14:52:13.496882 1168 raft_consensus.cc:357] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.497061 1168 raft_consensus.cc:383] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.497112 1168 raft_consensus.cc:738] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.497283 1168 consensus_queue.cc:260] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:13.497748 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f99d417ef074f978a91236d31552315 consensus state has no leader
I20250115 14:52:13.497776 1168 ts_tablet_manager.cc:1428] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:13.498039 1168 tablet_bootstrap.cc:492] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:13.498308 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f1e3879802464acaa279dfe4fbb634bb
I20250115 14:52:13.499128 743 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e3879802464acaa279dfe4fbb634bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.499578 1168 tablet_bootstrap.cc:654] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.500339 1166 ts_tablet_manager.cc:1918] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.500445 1166 log.cc:1198] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b5177265ba974ad6a381adb1e8c49589
I20250115 14:52:13.500697 1166 ts_tablet_manager.cc:1939] T b5177265ba974ad6a381adb1e8c49589 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.501595 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b5177265ba974ad6a381adb1e8c49589 (table default.Table_b6a520bf69264c44b1da30c0ffb41cc0 [id=f0b642abaec54506b98190fd1ac9ce67]) successfully deleted
I20250115 14:52:13.503118 1162 tablet_bootstrap.cc:492] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:13.503787 1162 ts_tablet_manager.cc:1397] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.000s
I20250115 14:52:13.508942 1162 raft_consensus.cc:357] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.509478 1164 tablet_bootstrap.cc:492] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.509647 1166 tablet_replica.cc:331] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.509690 1164 ts_tablet_manager.cc:1397] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20250115 14:52:13.510277 1162 raft_consensus.cc:383] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.510581 1164 raft_consensus.cc:357] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.510746 1162 raft_consensus.cc:738] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.510826 1164 raft_consensus.cc:383] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.510886 1164 raft_consensus.cc:738] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.510612 1751 tablet_replica.cc:331] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:13.511075 1164 consensus_queue.cc:260] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.509855 1166 raft_consensus.cc:2238] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.511472 1162 consensus_queue.cc:260] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.511544 1164 ts_tablet_manager.cc:1428] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:13.512024 1164 tablet_bootstrap.cc:492] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.513906 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ded663ac1197454bbde3c4c2d526002d (table default.Table_ca5ad5415b9144e583b4c771d24d4431 [id=a9793eda46fd4d07963603331438decb]) successfully deleted
I20250115 14:52:13.514003 1164 tablet_bootstrap.cc:654] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.514318 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a48a0af9422f43cf86531665c0fa1aee (table default.Table_c6d900c973594576916cf9d2b3a730b0 [id=e6815e32e72f46dd95661c2e68069d5e]) successfully deleted
I20250115 14:52:13.516623 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 4294c49da1e044be993100b3cf5e2cea with delete_type TABLET_DATA_DELETED (replaced by tablet d1259256b7744e739581125611abbe67) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.517597 1166 raft_consensus.cc:2267] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.511708 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet c497fb29752d4ed7bda207301f90cf05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.512190 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.512573 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2d55a4e32448fbb98f704dbf1657b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.513007 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet e6821f4c34f042c8b4819176fe4418ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.513644 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e3879802464acaa279dfe4fbb634bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.517900 1751 raft_consensus.cc:2238] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.519094 1751 raft_consensus.cc:2267] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:13.519642 984 consensus_peers.cc:487] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:13.519788 1162 ts_tablet_manager.cc:1428] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20250115 14:52:13.521342 1162 tablet_bootstrap.cc:492] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:13.523495 1162 tablet_bootstrap.cc:654] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.528002 1167 ts_tablet_manager.cc:1918] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.531651 1166 ts_tablet_manager.cc:1905] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.532378 1167 log.cc:1198] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ba596cfb3ce64718a42a9083975edc8d
I20250115 14:52:13.532770 1167 ts_tablet_manager.cc:1939] T ba596cfb3ce64718a42a9083975edc8d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.533231 1167 tablet_replica.cc:331] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.533489 1167 raft_consensus.cc:2238] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.533675 1167 raft_consensus.cc:2267] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.534215 1167 ts_tablet_manager.cc:1905] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.536679 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a6f3f8cae6244968927b864e7ce97565 (table default.Table_5d9f72946dd846b98daa1df0c392cb72 [id=02f9f84c80334c36801ef8e760120ab3]). Expected version 2 got 0
I20250115 14:52:13.540184 453 master_service.cc:946] ReplaceTablet: received request to replace tablet 356a0dfd7bb746548e0fb5b6a0935318 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:13.540220 1167 ts_tablet_manager.cc:1918] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:13.541522 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 115868us
W20250115 14:52:13.543468 848 leader_election.cc:343] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.543591 848 leader_election.cc:304] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.543852 2657 consensus_queue.cc:1035] T ec8c34919648447fac14208ede0ddc01 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:13.544346 2657 raft_consensus.cc:2743] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:13.544031 848 leader_election.cc:343] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.544765 848 leader_election.cc:304] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.545001 2549 raft_consensus.cc:2743] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:13.545329 848 leader_election.cc:343] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.545708 848 leader_election.cc:304] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.546103 2549 raft_consensus.cc:2743] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.548261 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 4294c49da1e044be993100b3cf5e2cea (table default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]) successfully deleted
I20250115 14:52:13.551026 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 3b82924e735846b2914520684d1b8fc7 (table default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]) successfully deleted
I20250115 14:52:13.551319 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet ba596cfb3ce64718a42a9083975edc8d (table default.Table_e1110e8f11bb4423bcdd8fedb129e390 [id=4d98519fb7194978a2a1533d7e42f2f1]) successfully deleted
I20250115 14:52:13.521685 1751 ts_tablet_manager.cc:1905] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.526362 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet ed777d7496514ec5986a7e6f53d532b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.537962 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f23815196047628d562da7312b010f with delete_type TABLET_DATA_DELETED (replaced by tablet 05bb99f36ba649d8b607e9a95ab58ed0) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.556425 1168 tablet_bootstrap.cc:492] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.556614 1168 ts_tablet_manager.cc:1397] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.001s
I20250115 14:52:13.557387 1168 raft_consensus.cc:357] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.557636 1168 raft_consensus.cc:383] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.557719 1168 raft_consensus.cc:738] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.557950 1168 consensus_queue.cc:260] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.558878 1168 ts_tablet_manager.cc:1428] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:13.559178 1168 tablet_bootstrap.cc:492] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:13.560930 1168 tablet_bootstrap.cc:654] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.562950 881 tablet_service.cc:1514] Processing DeleteTablet for tablet e6821f4c34f042c8b4819176fe4418ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.563441 1166 ts_tablet_manager.cc:1918] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.563539 1166 log.cc:1198] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ca128bf3d6fc430298da932a78f54184
I20250115 14:52:13.563797 1166 ts_tablet_manager.cc:1939] T ca128bf3d6fc430298da932a78f54184 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.565649 883 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e3879802464acaa279dfe4fbb634bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.566269 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.566388 2549 raft_consensus.cc:491] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.566515 2549 raft_consensus.cc:513] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.567116 2549 leader_election.cc:290] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:13.568454 848 consensus_peers.cc:487] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:13.569053 848 consensus_peers.cc:487] T dc363f70f3f44d309f69fac9356c6257 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:13.566608 882 tablet_service.cc:1514] Processing DeleteTablet for tablet c497fb29752d4ed7bda207301f90cf05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.570948 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_65ffb14a8a6d47a48387d591aae1370d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.571897 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65ffb14a8a6d47a48387d591aae1370d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:13.572078 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 45304.1us
W20250115 14:52:13.573451 847 leader_election.cc:343] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: a48227ec5aa846c3ac63bd29ed38be25
I20250115 14:52:13.576597 1166 tablet_replica.cc:331] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.576524 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ca128bf3d6fc430298da932a78f54184 (table default.Table_8f53bbc08e5340aab38bb0ce8ef7aa16 [id=6cdcb9c1b34642d8a792807746ac26f5]) successfully deleted
W20250115 14:52:13.579000 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c497fb29752d4ed7bda207301f90cf05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c497fb29752d4ed7bda207301f90cf05 already in progress: creating tablet
W20250115 14:52:13.579254 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 already in progress: creating tablet
W20250115 14:52:13.579460 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e6821f4c34f042c8b4819176fe4418ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6821f4c34f042c8b4819176fe4418ff already in progress: creating tablet
I20250115 14:52:13.580040 1168 tablet_bootstrap.cc:492] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.580200 1168 ts_tablet_manager.cc:1397] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20250115 14:52:13.580869 1168 raft_consensus.cc:357] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.581092 1168 raft_consensus.cc:383] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.581171 1168 raft_consensus.cc:738] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.581406 1168 consensus_queue.cc:260] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.581924 1168 ts_tablet_manager.cc:1428] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:13.582204 1168 tablet_bootstrap.cc:492] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:13.583712 1168 tablet_bootstrap.cc:654] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.580403 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48227ec5aa846c3ac63bd29ed38be25" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.584143 1166 raft_consensus.cc:2238] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.585199 1166 raft_consensus.cc:2267] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.581321 1164 tablet_bootstrap.cc:492] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.585757 1164 ts_tablet_manager.cc:1397] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.003s
I20250115 14:52:13.583874 1167 log.cc:1198] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/6ce498ffd3f04c5d8eb642739a838903
I20250115 14:52:13.587118 1167 ts_tablet_manager.cc:1939] T 6ce498ffd3f04c5d8eb642739a838903 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.587564 1167 tablet_replica.cc:331] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.587606 1164 raft_consensus.cc:357] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.587798 1167 raft_consensus.cc:2238] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.587854 1164 raft_consensus.cc:383] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.587919 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 6ce498ffd3f04c5d8eb642739a838903 (table default.Table_35aa379f66554bac80e34f11e41062c7 [id=c91058c4abd64f2fa0dadce63f6ce0f0]) successfully deleted
I20250115 14:52:13.589138 1162 tablet_bootstrap.cc:492] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:13.589313 1162 ts_tablet_manager.cc:1397] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.001s
I20250115 14:52:13.590420 1162 raft_consensus.cc:357] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.590667 1162 raft_consensus.cc:383] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.590772 1162 raft_consensus.cc:738] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.591043 1162 consensus_queue.cc:260] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:13.592967 848 consensus_peers.cc:487] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5ffa997ca6c46559830b1dedf2d1a66. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:13.593187 848 leader_election.cc:343] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.593320 848 leader_election.cc:304] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.587939 1166 ts_tablet_manager.cc:1905] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.593808 1164 raft_consensus.cc:738] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.596752 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89ec0a5d68408c897f43ab39df62b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.597013 1164 consensus_queue.cc:260] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.598402 1162 ts_tablet_manager.cc:1428] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250115 14:52:13.603755 1162 tablet_bootstrap.cc:492] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:13.605372 1751 ts_tablet_manager.cc:1918] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.607172 1168 tablet_bootstrap.cc:492] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.606434 2563 raft_consensus.cc:491] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.607316 1168 ts_tablet_manager.cc:1397] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.000s
I20250115 14:52:13.607358 1751 log.cc:1198] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e244689cef6844e2935becd265eecad0
I20250115 14:52:13.607391 2563 raft_consensus.cc:513] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.607646 1751 ts_tablet_manager.cc:1939] T e244689cef6844e2935becd265eecad0 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:13.607967 2563 leader_election.cc:290] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.599678 1166 ts_tablet_manager.cc:1918] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.608392 1166 log.cc:1198] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/580742eb5dac4dba9995809890e0e315
I20250115 14:52:13.608387 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1121) took 1013 ms. Trace:
I20250115 14:52:13.608460 1751 rpcz_store.cc:276] 0115 14:52:12.594659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:12.612422 (+ 17763us) service_pool.cc:225] Handling call
0115 14:52:13.510566 (+898144us) ts_tablet_manager.cc:1145] Deleting tablet e244689cef6844e2935becd265eecad0
0115 14:52:13.510575 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.581301 (+ 70726us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.605277 (+ 23976us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.608368 (+ 3091us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":897995}
I20250115 14:52:13.608692 1166 ts_tablet_manager.cc:1939] T 580742eb5dac4dba9995809890e0e315 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.608793 1751 tablet_replica.cc:331] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:13.608875 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb34bf9a3464798b9f28c50d6fc11" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:13.608958 1751 raft_consensus.cc:2238] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.609064 909 raft_consensus.cc:2463] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:13.609115 1751 raft_consensus.cc:2267] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.609694 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 580742eb5dac4dba9995809890e0e315 (table default.Table_d8d3e6a881ce475a8f1319238ab8c610 [id=b6de0dbbced44c83af5f98a9d1020cfc]) successfully deleted
I20250115 14:52:13.609860 983 leader_election.cc:304] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:13.610005 1164 ts_tablet_manager.cc:1428] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20250115 14:52:13.610234 1167 raft_consensus.cc:2267] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.610917 2668 raft_consensus.cc:2743] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.610924 1167 ts_tablet_manager.cc:1905] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.611541 2563 raft_consensus.cc:2798] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:13.611613 2563 raft_consensus.cc:491] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:13.611663 2563 raft_consensus.cc:3054] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:13.614264 2563 raft_consensus.cc:513] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.615319 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb34bf9a3464798b9f28c50d6fc11" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.615355 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb34bf9a3464798b9f28c50d6fc11" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:13.615494 909 raft_consensus.cc:3054] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:13.616171 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f1e3879802464acaa279dfe4fbb634bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e3879802464acaa279dfe4fbb634bb already in progress: creating tablet
I20250115 14:52:13.617631 1168 raft_consensus.cc:357] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.617799 1168 raft_consensus.cc:383] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.617827 909 raft_consensus.cc:2463] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:13.617847 1168 raft_consensus.cc:738] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.618026 1168 consensus_queue.cc:260] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.618132 1167 ts_tablet_manager.cc:1918] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:13.618219 1167 log.cc:1198] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/97b599c6b8174959b8b66e90e9242057
I20250115 14:52:13.618391 983 leader_election.cc:304] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:13.618502 1168 ts_tablet_manager.cc:1428] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250115 14:52:13.618544 1167 ts_tablet_manager.cc:1939] T 97b599c6b8174959b8b66e90e9242057 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.618633 2273 raft_consensus.cc:2798] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:13.618731 2273 raft_consensus.cc:695] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:13.618749 1168 tablet_bootstrap.cc:492] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:13.618930 2273 consensus_queue.cc:237] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.619060 1162 tablet_bootstrap.cc:654] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.619467 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 97b599c6b8174959b8b66e90e9242057 (table default.Table_e3e5361d9cb144999d02ff4df3e1999b [id=5063a3c3ef3a4b1794a2c8532ab7adf6]) successfully deleted
I20250115 14:52:13.620029 1168 tablet_bootstrap.cc:654] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.620162 1167 tablet_replica.cc:331] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.621189 1167 raft_consensus.cc:2238] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:13.621822 984 leader_election.cc:343] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.622170 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b82924e735846b2914520684d1b8fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.622267 1166 tablet_replica.cc:331] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.622269 1167 raft_consensus.cc:2267] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.622426 1166 raft_consensus.cc:2238] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.622560 1166 raft_consensus.cc:2267] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.622710 1167 ts_tablet_manager.cc:1905] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.623133 1166 ts_tablet_manager.cc:1905] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.623133 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb34bf9a3464798b9f28c50d6fc11" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:13.623723 984 leader_election.cc:343] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.625092 1164 tablet_bootstrap.cc:492] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.626610 1164 tablet_bootstrap.cc:654] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.627347 1751 ts_tablet_manager.cc:1905] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.632769 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a6f3f8cae6244968927b864e7ce97565 (table default.Table_5d9f72946dd846b98daa1df0c392cb72 [id=02f9f84c80334c36801ef8e760120ab3]). Expected version 2 got 0
I20250115 14:52:13.633186 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 5df0250d08214a9783c392071126ed4a (table default.Table_90c834d5ef634e6aa797dd220e3d5faa [id=8ffbe34c2a11417f8ca50dbdd627bf22]). Expected version 2 got 0
I20250115 14:52:13.634879 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 0f82d944a7e04ace8e4addb3897d2c8e
I20250115 14:52:13.635367 451 catalog_manager.cc:6596] ReplaceTablet: tablet 0f82d944a7e04ace8e4addb3897d2c8e deleted and replaced by tablet cda6870240d24eedaf9ddf57794b880e
I20250115 14:52:13.635795 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f82d944a7e04ace8e4addb3897d2c8e with delete_type TABLET_DATA_DELETED (replaced by tablet cda6870240d24eedaf9ddf57794b880e) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.637463 2563 leader_election.cc:290] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.637545 1168 tablet_bootstrap.cc:492] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.637676 1168 ts_tablet_manager.cc:1397] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250115 14:52:13.638314 1168 raft_consensus.cc:357] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.638499 1168 raft_consensus.cc:383] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.638541 1168 raft_consensus.cc:738] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.638710 1168 consensus_queue.cc:260] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.639190 1168 ts_tablet_manager.cc:1428] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250115 14:52:13.639345 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 23318b1ad0774a76880def17a64ac059
I20250115 14:52:13.639441 1168 tablet_bootstrap.cc:492] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:13.640053 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 23318b1ad0774a76880def17a64ac059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.640071 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 356a0dfd7bb746548e0fb5b6a0935318
I20250115 14:52:13.640515 453 catalog_manager.cc:6596] ReplaceTablet: tablet 356a0dfd7bb746548e0fb5b6a0935318 deleted and replaced by tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc
I20250115 14:52:13.640867 1168 tablet_bootstrap.cc:654] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.641759 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 356a0dfd7bb746548e0fb5b6a0935318 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.641916 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f82d944a7e04ace8e4addb3897d2c8e with delete_type TABLET_DATA_DELETED (replaced by tablet cda6870240d24eedaf9ddf57794b880e) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.642501 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 23318b1ad0774a76880def17a64ac059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.643237 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 356a0dfd7bb746548e0fb5b6a0935318 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.646759 1166 ts_tablet_manager.cc:1918] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.649662 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 23318b1ad0774a76880def17a64ac059 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.649847 881 tablet_service.cc:1514] Processing DeleteTablet for tablet 356a0dfd7bb746548e0fb5b6a0935318 with delete_type TABLET_DATA_DELETED (replaced by tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.652621 1166 log.cc:1198] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/4294c49da1e044be993100b3cf5e2cea
I20250115 14:52:13.652949 1166 ts_tablet_manager.cc:1939] T 4294c49da1e044be993100b3cf5e2cea P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.653563 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_06ae3b8ba32f4b05af77d455a746ecd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.653805 1166 tablet_replica.cc:331] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.653982 1166 raft_consensus.cc:2238] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.653970 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 4294c49da1e044be993100b3cf5e2cea (table default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]) successfully deleted
I20250115 14:52:13.654129 1166 raft_consensus.cc:2267] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:13.654150 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ae3b8ba32f4b05af77d455a746ecd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.655010 1168 tablet_bootstrap.cc:492] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.655143 1168 ts_tablet_manager.cc:1397] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250115 14:52:13.655402 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_24d51c1e7d8d4fdd83f72d6094af1d6b" } new_table_name: "default.Table_342273ba4aae48e6a1ac2d15b0a11de2" modify_external_catalogs: true
I20250115 14:52:13.655827 1168 raft_consensus.cc:357] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.656009 1168 raft_consensus.cc:383] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.656066 1168 raft_consensus.cc:738] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.656235 1168 consensus_queue.cc:260] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.656814 1168 ts_tablet_manager.cc:1428] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:13.657191 1168 tablet_bootstrap.cc:492] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:13.657274 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_01627515f7f64b1eb74dcf308bc61ea6" } modify_external_catalogs: true
I20250115 14:52:13.657425 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_01627515f7f64b1eb74dcf308bc61ea6" } modify_external_catalogs: true
I20250115 14:52:13.658591 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cc773ad5c57c4058baad0482e8dd2b79" } modify_external_catalogs: true
I20250115 14:52:13.658677 1168 tablet_bootstrap.cc:654] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.658759 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cc773ad5c57c4058baad0482e8dd2b79" } modify_external_catalogs: true
I20250115 14:52:13.661294 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7584e090790949b3b7d23a1ced5078be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.661780 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7584e090790949b3b7d23a1ced5078be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.662837 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_554ebf7919a940628afd939485c575d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.663317 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_554ebf7919a940628afd939485c575d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.664377 2656 tablet.cc:1721] T a6f3f8cae6244968927b864e7ce97565 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:13.664505 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f0e1896dc4ac463398d14176e3deda35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.664932 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0e1896dc4ac463398d14176e3deda35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.665756 2655 tablet.cc:1721] T a6f3f8cae6244968927b864e7ce97565 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:13.667783 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 50f8d93d655a447cb9e92997342a61a7
I20250115 14:52:13.668476 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_32d4e94660334e308afef5563b749191" } modify_external_catalogs: true
I20250115 14:52:13.668643 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_32d4e94660334e308afef5563b749191" } modify_external_catalogs: true
I20250115 14:52:13.669240 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 50f8d93d655a447cb9e92997342a61a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.669407 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_91a2412f46784a6d870afa172bf3849e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.669844 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91a2412f46784a6d870afa172bf3849e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.671101 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f82d944a7e04ace8e4addb3897d2c8e with delete_type TABLET_DATA_DELETED (replaced by tablet cda6870240d24eedaf9ddf57794b880e) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.671406 1166 ts_tablet_manager.cc:1905] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.672088 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 4294c49da1e044be993100b3cf5e2cea from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:13.673552 1167 ts_tablet_manager.cc:1918] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.673647 1167 log.cc:1198] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/65a6f83730f74d13823bd15d93538df7
I20250115 14:52:13.673906 1167 ts_tablet_manager.cc:1939] T 65a6f83730f74d13823bd15d93538df7 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.673974 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 50f8d93d655a447cb9e92997342a61a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.675819 1751 ts_tablet_manager.cc:1918] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:13.675920 1751 log.cc:1198] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/7233ebf3bae1462d8eeb3669b39944c0
I20250115 14:52:13.676221 1751 ts_tablet_manager.cc:1939] T 7233ebf3bae1462d8eeb3669b39944c0 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:13.676961 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1125) took 1018 ms. Trace:
W20250115 14:52:13.677038 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e4b8b77c11a4582b0d6b85f4c49aa82 consensus state has no leader
I20250115 14:52:13.678642 1167 tablet_replica.cc:331] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.678848 1167 raft_consensus.cc:2238] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.678869 1168 tablet_bootstrap.cc:492] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.678998 1168 ts_tablet_manager.cc:1397] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20250115 14:52:13.679813 1168 raft_consensus.cc:357] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.679996 1168 raft_consensus.cc:383] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.680047 1168 raft_consensus.cc:738] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.680652 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 50f8d93d655a447cb9e92997342a61a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.680929 1166 ts_tablet_manager.cc:1918] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.681036 1166 log.cc:1198] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/3b82924e735846b2914520684d1b8fc7
I20250115 14:52:13.681098 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 65a6f83730f74d13823bd15d93538df7 (table default.Table_8a0954ff11744933ba4752116830cdd7 [id=a3bf28673dac4ddab579709d6fd8885d]) successfully deleted
I20250115 14:52:13.680228 1168 consensus_queue.cc:260] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.681289 1166 ts_tablet_manager.cc:1939] T 3b82924e735846b2914520684d1b8fc7 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.681627 1168 ts_tablet_manager.cc:1428] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:52:13.681878 1168 tablet_bootstrap.cc:492] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:13.683295 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a300a76b8f7942dea648a20cbb3d5368 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a300a76b8f7942dea648a20cbb3d5368 already in progress: creating tablet
I20250115 14:52:13.683815 1168 tablet_bootstrap.cc:654] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:13.683965 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ac97cd8e11cf4b5e8978902d1c99aad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac97cd8e11cf4b5e8978902d1c99aad8 already in progress: creating tablet
W20250115 14:52:13.684180 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet cc9701156b0e48498666e34b3bec3043 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9701156b0e48498666e34b3bec3043 already in progress: creating tablet
I20250115 14:52:13.690086 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 93aaea90b0924327807e7f594154b14e
I20250115 14:52:13.690968 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b322d270a8f74d2e95bb772545cd9ade"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.697510 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
I20250115 14:52:13.677039 1751 rpcz_store.cc:276] 0115 14:52:12.658283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:12.658505 (+ 222us) service_pool.cc:225] Handling call
0115 14:52:13.608723 (+950218us) ts_tablet_manager.cc:1145] Deleting tablet 7233ebf3bae1462d8eeb3669b39944c0
0115 14:52:13.608757 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.639137 (+ 30380us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.675719 (+ 36582us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.676944 (+ 1225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":950031}
I20250115 14:52:13.698338 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ad988e25800f4af9a069b5a80582f749 (table default.Table_162eb94aa1674d21ae644021401a54e8 [id=e3887aa962874cfaa7ee89503ca1a49a]). Expected version 1 got 0
I20250115 14:52:13.698318 1167 raft_consensus.cc:2267] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.700232 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 7233ebf3bae1462d8eeb3669b39944c0 (table default.Table_1b5ca8c3048846588478dbfbebf1ff82 [id=8170a41205004f9ca83aa1c0ab7a2502]) successfully deleted
I20250115 14:52:13.693842 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 3b82924e735846b2914520684d1b8fc7 (table default.Table_d551e62017774245b9704a679dbd6e6b [id=c515f4c6803b44c08adffb4866ac46ba]) successfully deleted
W20250115 14:52:13.701591 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0f82d944a7e04ace8e4addb3897d2c8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f82d944a7e04ace8e4addb3897d2c8e already in progress: creating tablet
W20250115 14:52:13.701776 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b322d270a8f74d2e95bb772545cd9ade in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:13.701822 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 23318b1ad0774a76880def17a64ac059 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23318b1ad0774a76880def17a64ac059 already in progress: creating tablet
W20250115 14:52:13.702085 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 356a0dfd7bb746548e0fb5b6a0935318 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 356a0dfd7bb746548e0fb5b6a0935318 already in progress: creating tablet
W20250115 14:52:13.702307 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 50f8d93d655a447cb9e92997342a61a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50f8d93d655a447cb9e92997342a61a7 already in progress: creating tablet
I20250115 14:52:13.703364 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 93aaea90b0924327807e7f594154b14e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.704790 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f6993d1e422647d59233c375c8894af3" } new_table_name: "default.Table_7e0b2774d08642179743e596fa507d42" modify_external_catalogs: true
I20250115 14:52:13.705034 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 93aaea90b0924327807e7f594154b14e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:13.705634 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 93aaea90b0924327807e7f594154b14e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93aaea90b0924327807e7f594154b14e already in progress: creating tablet
I20250115 14:52:13.706102 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b8c3f1232b1a40d0a1110de0eb3289a6" } modify_external_catalogs: true
I20250115 14:52:13.706280 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b8c3f1232b1a40d0a1110de0eb3289a6" } modify_external_catalogs: true
I20250115 14:52:13.707321 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 93aaea90b0924327807e7f594154b14e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.708173 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ec9e7c8e59aa4ea59bad35c8a7591a79
I20250115 14:52:13.708748 743 tablet_service.cc:1514] Processing DeleteTablet for tablet ec9e7c8e59aa4ea59bad35c8a7591a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:13.709292 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ec9e7c8e59aa4ea59bad35c8a7591a79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec9e7c8e59aa4ea59bad35c8a7591a79 already in progress: creating tablet
I20250115 14:52:13.709616 1751 tablet_replica.cc:331] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:13.709811 1751 raft_consensus.cc:2238] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.709944 1751 raft_consensus.cc:2267] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.710431 1751 ts_tablet_manager.cc:1905] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.712025 882 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.712076 1167 ts_tablet_manager.cc:1905] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.713914 452 catalog_manager.cc:5526] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:13.714349 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
I20250115 14:52:13.715183 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3962d1f342824bf990b968aeba430bff" } new_table_name: "default.Table_3ac7568d7b354403935f7eb4c63d23e8" modify_external_catalogs: true
I20250115 14:52:13.716439 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_349a30fe26984a52913598e0f22c9295" } new_table_name: "default.Table_141efb598ef84dfdace28d40f3843d86" modify_external_catalogs: true
W20250115 14:52:13.720546 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98137e1c6ffa4352af8abca6f8d5748d consensus state has no leader
I20250115 14:52:13.720927 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_56d30c076381477494fb1b08ef4d4c46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.721464 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56d30c076381477494fb1b08ef4d4c46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.722882 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_768b0287db3a4e8c9908083f55f84209" } new_table_name: "default.Table_6c72bc6feb014b768d58cf8c6095fa8a" modify_external_catalogs: true
I20250115 14:52:13.726300 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7cf17435b88c4e87800daefcd4fa35ec
I20250115 14:52:13.727088 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf17435b88c4e87800daefcd4fa35ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.727039 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4003728871534ed4b55a545f2cf5821f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.727617 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4003728871534ed4b55a545f2cf5821f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:13.727711 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7cf17435b88c4e87800daefcd4fa35ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cf17435b88c4e87800daefcd4fa35ec already in progress: creating tablet
W20250115 14:52:13.729319 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e459144038d411d9e76322dee8f004b consensus state has no leader
W20250115 14:52:13.730353 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59e5cc832ffa4740aaacd65a565420f5 consensus state has no leader
I20250115 14:52:13.730867 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_5d9f72946dd846b98daa1df0c392cb72 [id=02f9f84c80334c36801ef8e760120ab3] alter complete (version 2)
I20250115 14:52:13.738907 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet ec9e7c8e59aa4ea59bad35c8a7591a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.739274 882 tablet_service.cc:1514] Processing DeleteTablet for tablet ec9e7c8e59aa4ea59bad35c8a7591a79 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.739871 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf17435b88c4e87800daefcd4fa35ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.740379 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf17435b88c4e87800daefcd4fa35ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:13.742702 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37970a1f9073400ebf973c4ac3510a72 has no consensus state
I20250115 14:52:13.743920 882 tablet_service.cc:1514] Processing DeleteTablet for tablet b76d1bbf90c6416ca60805c9d8053bd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e10fe1445be457a8c848c30771af866) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:13.743798 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 6afbdd0b585e4e7e82067dce061924d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6afbdd0b585e4e7e82067dce061924d0 already in progress: creating tablet
I20250115 14:52:13.745384 2669 raft_consensus.cc:491] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.745416 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b82924e735846b2914520684d1b8fc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.745532 2668 raft_consensus.cc:491] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.745502 2669 raft_consensus.cc:513] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.745630 2668 raft_consensus.cc:513] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.746090 2669 leader_election.cc:290] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:13.746593 882 tablet_service.cc:1467] Processing CreateTablet for tablet 8e312832ca9a43a3bcdc82863a2aeffd (DEFAULT_TABLE table=default.Table_221fbc5ac86b48eabef5ca465cce428d [id=d7bd90d0190249e38c12f3cd5fe9eaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.746614 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298fdd0217e4df6bfbafc49b2f2c65c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.746701 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e616d78933e4f84b9ec2acc51450b18" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.746953 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e312832ca9a43a3bcdc82863a2aeffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.746104 2668 leader_election.cc:290] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:13.747328 848 leader_election.cc:343] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.747401 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298fdd0217e4df6bfbafc49b2f2c65c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:52:13.747478 848 leader_election.cc:343] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.747570 1047 raft_consensus.cc:2463] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
W20250115 14:52:13.748045 847 leader_election.cc:343] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6e616d78933e4f84b9ec2acc51450b18
I20250115 14:52:13.748131 847 leader_election.cc:304] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:13.748188 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3b82924e735846b2914520684d1b8fc7 because the tablet was not found. No further retry: Not found: Tablet not found: 3b82924e735846b2914520684d1b8fc7
I20250115 14:52:13.748319 743 tablet_service.cc:1467] Processing CreateTablet for tablet 8e312832ca9a43a3bcdc82863a2aeffd (DEFAULT_TABLE table=default.Table_221fbc5ac86b48eabef5ca465cce428d [id=d7bd90d0190249e38c12f3cd5fe9eaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.748473 847 leader_election.cc:304] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.748656 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e312832ca9a43a3bcdc82863a2aeffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.748968 2341 raft_consensus.cc:2743] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.749123 2341 raft_consensus.cc:2798] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:13.749172 2341 raft_consensus.cc:491] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:13.749219 2341 raft_consensus.cc:3054] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:13.751557 2341 raft_consensus.cc:513] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.752090 2341 leader_election.cc:290] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:13.753031 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298fdd0217e4df6bfbafc49b2f2c65c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:13.753190 1047 raft_consensus.cc:3054] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:13.753301 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 8e312832ca9a43a3bcdc82863a2aeffd (DEFAULT_TABLE table=default.Table_221fbc5ac86b48eabef5ca465cce428d [id=d7bd90d0190249e38c12f3cd5fe9eaac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.753640 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e312832ca9a43a3bcdc82863a2aeffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.755463 1047 raft_consensus.cc:2463] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:13.755812 847 leader_election.cc:304] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:13.756032 2668 raft_consensus.cc:2798] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:13.756114 2668 raft_consensus.cc:695] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:13.756323 2668 consensus_queue.cc:237] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.757714 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298fdd0217e4df6bfbafc49b2f2c65c" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:13.758467 848 leader_election.cc:343] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.761633 771 raft_consensus.cc:3054] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:13.764815 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8bfb65359dea4a2e810cb2827a3161a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.766546 1751 ts_tablet_manager.cc:1918] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.766655 1751 log.cc:1198] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/0fd94e4a7fef44f5be462e7a77e8a123
I20250115 14:52:13.766960 1751 ts_tablet_manager.cc:1939] T 0fd94e4a7fef44f5be462e7a77e8a123 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:13.767781 1751 tablet_replica.cc:331] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:13.767972 1751 raft_consensus.cc:2238] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.767935 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 0fd94e4a7fef44f5be462e7a77e8a123 (table default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]) successfully deleted
I20250115 14:52:13.768108 1751 raft_consensus.cc:2267] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.768671 1167 ts_tablet_manager.cc:1918] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.768802 1167 log.cc:1198] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/076278b53996407c8a5c73a389459eaa
I20250115 14:52:13.769754 1164 tablet_bootstrap.cc:492] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.769929 1164 ts_tablet_manager.cc:1397] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.145s user 0.002s sys 0.001s
I20250115 14:52:13.770087 743 tablet_service.cc:1467] Processing CreateTablet for tablet 27fc79ad43f14002a6fe1502a4347ea7 (DEFAULT_TABLE table=default.Table_924719cc6f6a45fc99e425268591354c [id=b085d42488df4fee9c6da3f98b0a163a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.770463 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27fc79ad43f14002a6fe1502a4347ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.770488 882 tablet_service.cc:1467] Processing CreateTablet for tablet 27fc79ad43f14002a6fe1502a4347ea7 (DEFAULT_TABLE table=default.Table_924719cc6f6a45fc99e425268591354c [id=b085d42488df4fee9c6da3f98b0a163a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.770841 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27fc79ad43f14002a6fe1502a4347ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.771411 1164 raft_consensus.cc:357] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.771659 1164 raft_consensus.cc:383] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.771754 1164 raft_consensus.cc:738] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.771997 1164 consensus_queue.cc:260] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.772794 1164 ts_tablet_manager.cc:1428] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:13.773079 1164 tablet_bootstrap.cc:492] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.774948 1164 tablet_bootstrap.cc:654] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.776571 1162 tablet_bootstrap.cc:492] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:13.776741 1168 tablet_bootstrap.cc:492] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:13.776799 1162 ts_tablet_manager.cc:1397] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.173s user 0.004s sys 0.012s
I20250115 14:52:13.776973 1168 ts_tablet_manager.cc:1397] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.009s
I20250115 14:52:13.777474 1162 raft_consensus.cc:357] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.777863 1162 raft_consensus.cc:383] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.777911 1162 raft_consensus.cc:738] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.778113 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 27fc79ad43f14002a6fe1502a4347ea7 (DEFAULT_TABLE table=default.Table_924719cc6f6a45fc99e425268591354c [id=b085d42488df4fee9c6da3f98b0a163a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:13.778496 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27fc79ad43f14002a6fe1502a4347ea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:13.778120 1162 consensus_queue.cc:260] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.779389 1162 ts_tablet_manager.cc:1428] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250115 14:52:13.781849 1162 tablet_bootstrap.cc:492] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:13.783216 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5e22d90ab7004e6fb221c1e26c5f564f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.783643 1162 tablet_bootstrap.cc:654] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:13.783739 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e22d90ab7004e6fb221c1e26c5f564f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.785284 447 master_service.cc:946] ReplaceTablet: received request to replace tablet abe3c66176664fa99c7a1fb480b56cca from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:13.786209 1164 tablet_bootstrap.cc:492] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.788656 1164 ts_tablet_manager.cc:1397] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20250115 14:52:13.789853 1164 raft_consensus.cc:357] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.790387 1751 ts_tablet_manager.cc:1905] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.793047 1164 raft_consensus.cc:383] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.793494 1164 raft_consensus.cc:738] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.793815 1164 consensus_queue.cc:260] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.794387 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet abe3c66176664fa99c7a1fb480b56cca
I20250115 14:52:13.794872 447 catalog_manager.cc:6596] ReplaceTablet: tablet abe3c66176664fa99c7a1fb480b56cca deleted and replaced by tablet a0e115eb01814dcbb4f91c369a1c6835
I20250115 14:52:13.795096 743 tablet_service.cc:1514] Processing DeleteTablet for tablet abe3c66176664fa99c7a1fb480b56cca with delete_type TABLET_DATA_DELETED (replaced by tablet a0e115eb01814dcbb4f91c369a1c6835) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.795830 1164 ts_tablet_manager.cc:1428] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:52:13.796078 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet abe3c66176664fa99c7a1fb480b56cca with delete_type TABLET_DATA_DELETED (replaced by tablet a0e115eb01814dcbb4f91c369a1c6835) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:13.796416 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet abe3c66176664fa99c7a1fb480b56cca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abe3c66176664fa99c7a1fb480b56cca already in progress: creating tablet
I20250115 14:52:13.796945 1164 tablet_bootstrap.cc:492] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.799216 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a97d102c174bc3a5e3a3c5762033b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.799857 2660 tablet.cc:1721] T 32eee3d3dfec464baf10c238babb9eb1 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:13.800096 1168 raft_consensus.cc:357] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.800277 1168 raft_consensus.cc:383] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.800346 1168 raft_consensus.cc:738] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.779650 1167 ts_tablet_manager.cc:1939] T 076278b53996407c8a5c73a389459eaa P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.780169 2669 raft_consensus.cc:491] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.803900 2669 raft_consensus.cc:513] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.795598 884 tablet_service.cc:1514] Processing DeleteTablet for tablet abe3c66176664fa99c7a1fb480b56cca with delete_type TABLET_DATA_DELETED (replaced by tablet a0e115eb01814dcbb4f91c369a1c6835) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.804620 2669 leader_election.cc:290] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:13.805176 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d850b85e948939008e5275d42e3cc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:13.805346 1047 raft_consensus.cc:2463] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:13.805711 847 leader_election.cc:304] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:13.806018 2669 raft_consensus.cc:2798] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:13.806087 2669 raft_consensus.cc:491] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:13.806143 2669 raft_consensus.cc:3054] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:13.808589 2669 raft_consensus.cc:513] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.809496 2669 leader_election.cc:290] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:13.809487 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d850b85e948939008e5275d42e3cc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.809790 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d850b85e948939008e5275d42e3cc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:13.809947 1047 raft_consensus.cc:3054] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:13.812783 848 leader_election.cc:343] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.812904 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4d850b85e948939008e5275d42e3cc" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:13.813084 1162 tablet_bootstrap.cc:492] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:13.813200 1162 ts_tablet_manager.cc:1397] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.002s
W20250115 14:52:13.813292 848 leader_election.cc:343] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.813895 1162 raft_consensus.cc:357] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.814047 1162 raft_consensus.cc:383] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.814092 1162 raft_consensus.cc:738] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.814296 1162 consensus_queue.cc:260] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.815088 1162 ts_tablet_manager.cc:1428] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:13.815243 1164 tablet_bootstrap.cc:654] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.816507 447 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 32eee3d3dfec464baf10c238babb9eb1 (table default.Table_a14212e9310040fa88dd2432f80fdc58 [id=31fde8bd2aa3452fb638c948eb5d8a55]). Expected version 1 got 0
I20250115 14:52:13.817348 1162 tablet_bootstrap.cc:492] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:13.815099 1047 raft_consensus.cc:2463] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:13.817816 847 leader_election.cc:304] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.818245 2669 raft_consensus.cc:2798] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:13.818457 2669 raft_consensus.cc:695] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:13.818691 2669 consensus_queue.cc:237] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.820096 1751 ts_tablet_manager.cc:1918] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.821496 1751 log.cc:1198] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/0cae4c9228c640e495507e8f7ad9f544
I20250115 14:52:13.821859 1751 ts_tablet_manager.cc:1939] T 0cae4c9228c640e495507e8f7ad9f544 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:13.824268 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1150) took 1003 ms. Trace:
I20250115 14:52:13.825039 455 catalog_manager.cc:5526] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:13.827170 1167 tablet_replica.cc:331] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.827358 1167 raft_consensus.cc:2238] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.827481 1167 raft_consensus.cc:2267] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.827919 1167 ts_tablet_manager.cc:1905] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.829133 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 076278b53996407c8a5c73a389459eaa (table default.Table_7a794e73718b4f888c401b3038c977d9 [id=2b96b49ab0d64a7db19e0fb7985b7bec]) successfully deleted
I20250115 14:52:13.832984 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce498ffd3f04c5d8eb642739a838903 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.833230 1166 tablet_replica.cc:331] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.833261 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 076278b53996407c8a5c73a389459eaa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:58 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.833400 1166 raft_consensus.cc:2238] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.833544 1166 raft_consensus.cc:2267] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:13.834126 441 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b -> Peer 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227): Couldn't send request to peer 1cc2e23c560f4096be7b14006bdfc525. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250115 14:52:13.834880 1166 ts_tablet_manager.cc:1905] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.835420 1751 rpcz_store.cc:276] 0115 14:52:12.820788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:12.831054 (+ 10266us) service_pool.cc:225] Handling call
0115 14:52:13.767725 (+936671us) ts_tablet_manager.cc:1145] Deleting tablet 0cae4c9228c640e495507e8f7ad9f544
0115 14:52:13.767735 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.807892 (+ 40157us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.820000 (+ 12108us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.824250 (+ 4250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":936579}
I20250115 14:52:13.840883 1166 ts_tablet_manager.cc:1918] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.840976 1166 log.cc:1198] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/6ce498ffd3f04c5d8eb642739a838903
I20250115 14:52:13.841241 1166 ts_tablet_manager.cc:1939] T 6ce498ffd3f04c5d8eb642739a838903 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.842077 1166 tablet_replica.cc:331] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.842237 1166 raft_consensus.cc:2238] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.842345 1166 raft_consensus.cc:2267] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.844720 1166 ts_tablet_manager.cc:1905] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.844642 1168 consensus_queue.cc:260] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.845187 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 6ce498ffd3f04c5d8eb642739a838903 (table default.Table_35aa379f66554bac80e34f11e41062c7 [id=c91058c4abd64f2fa0dadce63f6ce0f0]) successfully deleted
I20250115 14:52:13.847134 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d47eeb687f8046f2803dd6b6a5e6c153" } modify_external_catalogs: true
I20250115 14:52:13.847374 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d47eeb687f8046f2803dd6b6a5e6c153" } modify_external_catalogs: true
I20250115 14:52:13.848496 455 catalog_manager.cc:5526] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:13.833884 2273 raft_consensus.cc:491] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.853338 2273 raft_consensus.cc:513] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.853749 1751 tablet_replica.cc:331] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:13.853803 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 0cae4c9228c640e495507e8f7ad9f544 (table default.Table_dae457edf44a40a69a40030ae24a26ff [id=0dd965b7347c439d9abcc56446730332]) successfully deleted
I20250115 14:52:13.829692 1162 tablet_bootstrap.cc:654] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.853931 1751 raft_consensus.cc:2238] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.853956 2273 leader_election.cc:290] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:13.854028 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ac97cd8e11cf4b5e8978902d1c99aad8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac97cd8e11cf4b5e8978902d1c99aad8 already in progress: creating tablet
I20250115 14:52:13.854070 1751 raft_consensus.cc:2267] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:13.854218 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet cc9701156b0e48498666e34b3bec3043 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc9701156b0e48498666e34b3bec3043 already in progress: creating tablet
I20250115 14:52:13.854576 1751 ts_tablet_manager.cc:1905] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.856009 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2d55a4e32448fbb98f704dbf1657b9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:13.856556 984 leader_election.cc:343] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.857189 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2d55a4e32448fbb98f704dbf1657b9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:13.857659 983 leader_election.cc:343] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.857753 983 leader_election.cc:304] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.858183 2273 raft_consensus.cc:2743] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.861050 1168 ts_tablet_manager.cc:1428] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.062s user 0.000s sys 0.001s
I20250115 14:52:13.861337 1168 tablet_bootstrap.cc:492] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:13.865191 1166 ts_tablet_manager.cc:1918] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.851509 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f541e916d264738842cd965c7a2a5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.868505 1162 tablet_bootstrap.cc:492] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:13.868633 1162 ts_tablet_manager.cc:1397] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.001s
I20250115 14:52:13.869062 1168 tablet_bootstrap.cc:654] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.869336 1162 raft_consensus.cc:357] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.869524 1162 raft_consensus.cc:383] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.869578 1162 raft_consensus.cc:738] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.869798 1162 consensus_queue.cc:260] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.870946 1162 ts_tablet_manager.cc:1428] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:13.871109 1166 log.cc:1198] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/076278b53996407c8a5c73a389459eaa
I20250115 14:52:13.871387 1166 ts_tablet_manager.cc:1939] T 076278b53996407c8a5c73a389459eaa P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.872551 1162 tablet_bootstrap.cc:492] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:13.866784 1164 tablet_bootstrap.cc:492] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.872872 1164 ts_tablet_manager.cc:1397] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.002s
I20250115 14:52:13.872979 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4980e328dcce4600a482147b7a9c0b30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.873493 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4980e328dcce4600a482147b7a9c0b30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.873601 1164 raft_consensus.cc:357] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.873776 1164 raft_consensus.cc:383] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.873821 1164 raft_consensus.cc:738] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.875306 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d60ab02c55f94f3590d4b43fb3c8fa0a" } modify_external_catalogs: true
I20250115 14:52:13.875720 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 248cadcdef8b4e99923cdbca39690685 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.876382 1164 consensus_queue.cc:260] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.877107 1164 ts_tablet_manager.cc:1428] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250115 14:52:13.877403 1164 tablet_bootstrap.cc:492] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.877679 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 076278b53996407c8a5c73a389459eaa (table default.Table_7a794e73718b4f888c401b3038c977d9 [id=2b96b49ab0d64a7db19e0fb7985b7bec]) successfully deleted
I20250115 14:52:13.881428 1162 tablet_bootstrap.cc:654] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.881444 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbb6df0135694d7b83ed96983e487001" } new_table_name: "default.Table_b6ad5a9efbdf4507962e40f6fef3fb56" modify_external_catalogs: true
I20250115 14:52:13.881793 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b2b4a2f38c7444bd9638bc834e8bb2a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.882283 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2b4a2f38c7444bd9638bc834e8bb2a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.881439 1164 tablet_bootstrap.cc:654] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.888772 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5e15a5e365604ac2a0fc0ac6631a167f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.889288 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e15a5e365604ac2a0fc0ac6631a167f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.908999 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_214a95027bd94905b10b5c7285a31b00"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.909364 2273 raft_consensus.cc:491] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.909325 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4e5f5823275b40959b3058fbf28be96d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.909480 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_214a95027bd94905b10b5c7285a31b00 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.909502 2273 raft_consensus.cc:513] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:13.909709 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e5f5823275b40959b3058fbf28be96d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.910135 2273 leader_election.cc:290] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.910976 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89ec0a5d68408c897f43ab39df62b1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.911433 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89ec0a5d68408c897f43ab39df62b1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:13.911943 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_69ed5f68465549b38e7e27bf0d75eb2e" } modify_external_catalogs: true
I20250115 14:52:13.912087 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_69ed5f68465549b38e7e27bf0d75eb2e" } modify_external_catalogs: true
W20250115 14:52:13.912118 983 leader_election.cc:343] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:13.912416 984 leader_election.cc:343] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.912501 984 leader_election.cc:304] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.912588 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_81aa5a4246744ef69326175dae48e122 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 1 got 0
I20250115 14:52:13.912909 2273 raft_consensus.cc:2743] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.919967 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_57cca7a5af0b434b92b6e5c6eb5413f9" } modify_external_catalogs: true
I20250115 14:52:13.920120 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_57cca7a5af0b434b92b6e5c6eb5413f9" } modify_external_catalogs: true
I20250115 14:52:13.920164 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 4e459144038d411d9e76322dee8f004b from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:13.923611 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4003728871534ed4b55a545f2cf5821f" } new_table_name: "default.Table_1033f77f38414857a1255c05f93935d2" modify_external_catalogs: true
I20250115 14:52:13.924731 1164 tablet_bootstrap.cc:492] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.924957 1164 ts_tablet_manager.cc:1397] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20250115 14:52:13.925734 1164 raft_consensus.cc:357] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.925983 1164 raft_consensus.cc:383] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.926083 1164 raft_consensus.cc:738] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.926338 1164 consensus_queue.cc:260] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.931612 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 11aeb6c73b4d42058736ffce551e311c
I20250115 14:52:13.932870 1164 ts_tablet_manager.cc:1428] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250115 14:52:13.932946 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 11aeb6c73b4d42058736ffce551e311c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.932962 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 11aeb6c73b4d42058736ffce551e311c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.934288 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_55bce736d0234efa9eeee2a1745b955a" } new_table_name: "default.Table_24b52be733114df7b870252914287cb9" modify_external_catalogs: true
I20250115 14:52:13.935002 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 11aeb6c73b4d42058736ffce551e311c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.936048 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1f1b7f31ef374e49add057beb69dbbcf" } modify_external_catalogs: true
I20250115 14:52:13.936185 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1f1b7f31ef374e49add057beb69dbbcf" } modify_external_catalogs: true
I20250115 14:52:13.936851 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_81aa5a4246744ef69326175dae48e122" } new_table_name: "default.Table_1426afacf12444b5a8f8977fbac29d75" modify_external_catalogs: true
I20250115 14:52:13.937901 1164 tablet_bootstrap.cc:492] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:13.939545 1164 tablet_bootstrap.cc:654] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.941121 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet eb78eb2c5b9b44fba1a4e140a8e102d7
I20250115 14:52:13.942338 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet eb78eb2c5b9b44fba1a4e140a8e102d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.942580 884 tablet_service.cc:1514] Processing DeleteTablet for tablet eb78eb2c5b9b44fba1a4e140a8e102d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:13.943562 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 11aeb6c73b4d42058736ffce551e311c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11aeb6c73b4d42058736ffce551e311c already in progress: creating tablet
W20250115 14:52:13.944746 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afea664025e54f6099d825c8f7a0ea1a has no consensus state
I20250115 14:52:13.945082 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4e459144038d411d9e76322dee8f004b
I20250115 14:52:13.945439 452 catalog_manager.cc:6596] ReplaceTablet: tablet 4e459144038d411d9e76322dee8f004b deleted and replaced by tablet 6c68d7ac544c4606bacabc601f6675f7
I20250115 14:52:13.945780 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9ff76eab7cfc4904a16a9e1b4ff89486"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.946278 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff76eab7cfc4904a16a9e1b4ff89486 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.947269 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e459144038d411d9e76322dee8f004b with delete_type TABLET_DATA_DELETED (replaced by tablet 6c68d7ac544c4606bacabc601f6675f7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.947465 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e459144038d411d9e76322dee8f004b with delete_type TABLET_DATA_DELETED (replaced by tablet 6c68d7ac544c4606bacabc601f6675f7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.948432 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_54c4b2a5cfdc4d65b61e0bf4854a48b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:13.948590 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c62569d637274fd68dc1442fbe05f902"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:13.948889 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54c4b2a5cfdc4d65b61e0bf4854a48b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:13.948916 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c62569d637274fd68dc1442fbe05f902 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:13.948447 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bb95e5cd8bf543b79d33f685016648dd" } new_table_name: "default.Table_2ea7cdeffde54825be02af36704833a0" modify_external_catalogs: true
I20250115 14:52:13.952759 1751 ts_tablet_manager.cc:1918] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.939617 738 tablet_service.cc:1514] Processing DeleteTablet for tablet ba596cfb3ce64718a42a9083975edc8d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.956663 1751 log.cc:1198] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/a300a76b8f7942dea648a20cbb3d5368
I20250115 14:52:13.956987 1751 ts_tablet_manager.cc:1939] T a300a76b8f7942dea648a20cbb3d5368 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:13.957728 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1155) took 1026 ms. Trace:
I20250115 14:52:13.957818 1751 rpcz_store.cc:276] 0115 14:52:12.931060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:12.949099 (+ 18039us) service_pool.cc:225] Handling call
0115 14:52:13.853698 (+904599us) ts_tablet_manager.cc:1145] Deleting tablet a300a76b8f7942dea648a20cbb3d5368
0115 14:52:13.853705 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.902313 (+ 48608us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.952629 (+ 50316us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:13.957708 (+ 5079us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":904447}
I20250115 14:52:13.958218 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet a300a76b8f7942dea648a20cbb3d5368 (table default.Table_0034e4c667b44bbb83a5a93e321e5113 [id=f6bae7686ebc45488f5eb27a554d28a3]) successfully deleted
I20250115 14:52:13.944243 743 tablet_service.cc:1514] Processing DeleteTablet for tablet eb78eb2c5b9b44fba1a4e140a8e102d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.947963 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e459144038d411d9e76322dee8f004b with delete_type TABLET_DATA_DELETED (replaced by tablet 6c68d7ac544c4606bacabc601f6675f7) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:13.961792 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet eb78eb2c5b9b44fba1a4e140a8e102d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb78eb2c5b9b44fba1a4e140a8e102d7 already in progress: creating tablet
W20250115 14:52:13.962013 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4e459144038d411d9e76322dee8f004b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e459144038d411d9e76322dee8f004b already in progress: creating tablet
W20250115 14:52:13.965586 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75ccc1f31c6a4d2cba65324b7a71c2ce consensus state has no leader
I20250115 14:52:13.966770 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d36fa3146f0f40a59e5f96cce203fee0
I20250115 14:52:13.966897 1751 tablet_replica.cc:331] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:13.967080 1751 raft_consensus.cc:2238] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:13.967852 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet d36fa3146f0f40a59e5f96cce203fee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.968139 884 tablet_service.cc:1514] Processing DeleteTablet for tablet d36fa3146f0f40a59e5f96cce203fee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.968433 1751 raft_consensus.cc:2267] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.968981 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d60ab02c55f94f3590d4b43fb3c8fa0a" } modify_external_catalogs: true
I20250115 14:52:13.969715 2273 raft_consensus.cc:491] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.969714 741 tablet_service.cc:1514] Processing DeleteTablet for tablet d36fa3146f0f40a59e5f96cce203fee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.969831 2273 raft_consensus.cc:513] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.970402 2273 leader_election.cc:290] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.971480 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a97d102c174bc3a5e3a3c5762033b3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:13.972123 983 leader_election.cc:343] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.972915 1164 tablet_bootstrap.cc:492] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:13.972986 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72a97d102c174bc3a5e3a3c5762033b3" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:13.973107 1164 ts_tablet_manager.cc:1397] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
W20250115 14:52:13.973433 984 leader_election.cc:343] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.973528 984 leader_election.cc:304] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.973858 1164 raft_consensus.cc:357] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.974102 1164 raft_consensus.cc:383] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:13.974107 2273 raft_consensus.cc:2743] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.974215 1164 raft_consensus.cc:738] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:13.974484 1164 consensus_queue.cc:260] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:13.978271 1164 ts_tablet_manager.cc:1428] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:13.979036 1751 ts_tablet_manager.cc:1905] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.980604 1164 tablet_bootstrap.cc:492] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:13.981181 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a79f880415a4b16aabdbb7cd989efdb consensus state has no leader
I20250115 14:52:13.981521 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f06544890cbf460cbe218690e998eb1d
I20250115 14:52:13.980038 2273 raft_consensus.cc:491] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:13.981772 2273 raft_consensus.cc:513] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:13.982213 884 tablet_service.cc:1514] Processing DeleteTablet for tablet f06544890cbf460cbe218690e998eb1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:13.982340 2273 leader_election.cc:290] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:13.982620 1164 tablet_bootstrap.cc:654] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:13.983117 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f06544890cbf460cbe218690e998eb1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:13.983336 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f23815196047628d562da7312b010f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:13.983590 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d36fa3146f0f40a59e5f96cce203fee0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d36fa3146f0f40a59e5f96cce203fee0 already in progress: creating tablet
W20250115 14:52:13.983934 983 leader_election.cc:343] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.984571 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f23815196047628d562da7312b010f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:13.985280 984 leader_election.cc:343] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:13.985383 984 leader_election.cc:304] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:13.985808 2273 raft_consensus.cc:2743] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:13.986181 741 tablet_service.cc:1514] Processing DeleteTablet for tablet f06544890cbf460cbe218690e998eb1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:13.986380 1166 tablet_replica.cc:331] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.986546 1166 raft_consensus.cc:2238] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.986701 1166 raft_consensus.cc:2267] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.987206 1166 ts_tablet_manager.cc:1905] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.987485 448 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet a6f3f8cae6244968927b864e7ce97565 (table default.Table_5d9f72946dd846b98daa1df0c392cb72 [id=02f9f84c80334c36801ef8e760120ab3]). Expected version 2 got 0
I20250115 14:52:13.993546 1166 ts_tablet_manager.cc:1918] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:13.993647 1166 log.cc:1198] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/f06544890cbf460cbe218690e998eb1d
I20250115 14:52:13.994139 1166 ts_tablet_manager.cc:1939] T f06544890cbf460cbe218690e998eb1d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:13.994150 1167 ts_tablet_manager.cc:1918] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:13.994258 1167 log.cc:1198] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/a48227ec5aa846c3ac63bd29ed38be25
I20250115 14:52:13.994521 1167 ts_tablet_manager.cc:1939] T a48227ec5aa846c3ac63bd29ed38be25 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:13.994983 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet a48227ec5aa846c3ac63bd29ed38be25 (table default.Table_bb07dc16172246278fff8624127ab8bd [id=924019d3b4e842c2b3f173b307885151]) successfully deleted
W20250115 14:52:13.995712 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 1fc659e2d2d943a99aef530e7d71df88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fc659e2d2d943a99aef530e7d71df88 already in progress: creating tablet
I20250115 14:52:13.995977 1167 tablet_replica.cc:331] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:13.996430 1167 raft_consensus.cc:2238] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.996573 1167 raft_consensus.cc:2267] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.997048 1167 ts_tablet_manager.cc:1905] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:13.997220 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet f06544890cbf460cbe218690e998eb1d (table default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]) successfully deleted
I20250115 14:52:13.999068 452 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet a6f3f8cae6244968927b864e7ce97565 (table default.Table_5d9f72946dd846b98daa1df0c392cb72 [id=02f9f84c80334c36801ef8e760120ab3]). Expected version 2 got 0
I20250115 14:52:13.999123 1166 tablet_replica.cc:331] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:13.999280 1166 raft_consensus.cc:2238] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:13.999421 1166 raft_consensus.cc:2267] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:13.999907 1168 tablet_bootstrap.cc:492] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:14.000065 1168 ts_tablet_manager.cc:1397] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.139s user 0.000s sys 0.003s
I20250115 14:52:14.000195 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0c704df3a19a436e891ea2813ceb7db3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.000839 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c704df3a19a436e891ea2813ceb7db3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.000847 1168 raft_consensus.cc:357] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.001075 1168 raft_consensus.cc:383] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.001160 1168 raft_consensus.cc:738] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.006345 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f2993e1c95234a1db2ae98a8e08c43f8
I20250115 14:52:14.006947 1166 ts_tablet_manager.cc:1905] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.007194 884 tablet_service.cc:1514] Processing DeleteTablet for tablet f2993e1c95234a1db2ae98a8e08c43f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.007576 2669 raft_consensus.cc:491] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.007690 2669 raft_consensus.cc:513] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.007899 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f2993e1c95234a1db2ae98a8e08c43f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.008221 2669 leader_election.cc:290] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.008587 2273 raft_consensus.cc:491] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.008700 2273 raft_consensus.cc:513] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.009272 2273 leader_election.cc:290] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:14.009320 847 leader_election.cc:343] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: a2fe51b51994491ab38a644bfc55e3ac
I20250115 14:52:14.009775 771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2fe51b51994491ab38a644bfc55e3ac" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.009881 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6308f68e41554470b7d1321c60d4eee9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:14.010205 848 leader_election.cc:343] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.010295 848 leader_election.cc:304] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.010411 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6308f68e41554470b7d1321c60d4eee9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.010623 2669 raft_consensus.cc:2743] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:14.010874 984 leader_election.cc:343] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:14.011204 983 leader_election.cc:343] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.011286 983 leader_election.cc:304] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.011773 2273 raft_consensus.cc:2743] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.001408 1168 consensus_queue.cc:260] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.019258 2669 raft_consensus.cc:491] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.019392 2669 raft_consensus.cc:513] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.019868 1168 ts_tablet_manager.cc:1428] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20250115 14:52:14.019938 2669 leader_election.cc:290] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.020370 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52fb251b0a48c2a8a4e07d55d22d3f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:14.020563 1046 raft_consensus.cc:2463] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:14.021123 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52fb251b0a48c2a8a4e07d55d22d3f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.021248 847 leader_election.cc:304] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:52:14.021661 848 leader_election.cc:343] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.021951 2669 raft_consensus.cc:2798] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:14.022020 2669 raft_consensus.cc:491] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:14.022074 2669 raft_consensus.cc:3054] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.022053 1166 ts_tablet_manager.cc:1918] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.022145 1166 log.cc:1198] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ba596cfb3ce64718a42a9083975edc8d
I20250115 14:52:14.022408 1166 ts_tablet_manager.cc:1939] T ba596cfb3ce64718a42a9083975edc8d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:14.023283 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ba596cfb3ce64718a42a9083975edc8d (table default.Table_e1110e8f11bb4423bcdd8fedb129e390 [id=4d98519fb7194978a2a1533d7e42f2f1]) successfully deleted
I20250115 14:52:14.023968 738 tablet_service.cc:1514] Processing DeleteTablet for tablet f2993e1c95234a1db2ae98a8e08c43f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:13 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:14.025036 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet f2993e1c95234a1db2ae98a8e08c43f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2993e1c95234a1db2ae98a8e08c43f8 already in progress: creating tablet
I20250115 14:52:14.025034 2655 tablet.cc:1721] T 3e94ea76507f4646909ac0c1b2c3136a P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:14.025456 2669 raft_consensus.cc:513] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.026979 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52fb251b0a48c2a8a4e07d55d22d3f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:14.027354 2660 tablet.cc:1721] T 3e94ea76507f4646909ac0c1b2c3136a P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:14.027649 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b52fb251b0a48c2a8a4e07d55d22d3f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:14.027837 1046 raft_consensus.cc:3054] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.030651 1046 raft_consensus.cc:2463] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:14.031416 847 leader_election.cc:304] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:14.031669 2657 raft_consensus.cc:2798] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:14.031781 2657 raft_consensus.cc:695] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:14.032017 2657 consensus_queue.cc:237] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:14.033766 848 leader_election.cc:343] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.036187 2669 leader_election.cc:290] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.040710 2669 raft_consensus.cc:491] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.040841 2669 raft_consensus.cc:513] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.041394 2669 leader_election.cc:290] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.041920 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad988e25800f4af9a069b5a80582f749" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.042202 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad988e25800f4af9a069b5a80582f749" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:14.042449 1046 raft_consensus.cc:2463] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:14.042577 448 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_b6ad5a9efbdf4507962e40f6fef3fb56 [id=3903b654e8644b2ab335a5f98f106261] alter complete (version 1)
I20250115 14:52:14.043059 847 leader_election.cc:304] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:52:14.043440 848 leader_election.cc:343] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.043797 2669 raft_consensus.cc:2798] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:14.043947 2669 raft_consensus.cc:491] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:14.044096 2669 raft_consensus.cc:3054] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.046166 1162 tablet_bootstrap.cc:492] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.046298 1162 ts_tablet_manager.cc:1397] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.174s user 0.012s sys 0.022s
I20250115 14:52:14.047094 1162 raft_consensus.cc:357] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.047360 1162 raft_consensus.cc:383] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.047463 1162 raft_consensus.cc:738] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.047739 1162 consensus_queue.cc:260] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.048977 1162 ts_tablet_manager.cc:1428] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:14.049824 1162 tablet_bootstrap.cc:492] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.051708 2669 raft_consensus.cc:513] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.052829 1162 tablet_bootstrap.cc:654] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.052907 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad988e25800f4af9a069b5a80582f749" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:14.053460 848 leader_election.cc:343] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.058398 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 4adf1353bd554d8294a7f570931095d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.062947 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad988e25800f4af9a069b5a80582f749" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:14.063208 1046 raft_consensus.cc:3054] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.065729 1046 raft_consensus.cc:2463] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:14.066167 847 leader_election.cc:304] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.066581 2657 raft_consensus.cc:2798] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:14.066860 2657 raft_consensus.cc:695] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:14.067158 2657 consensus_queue.cc:237] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.071710 2669 leader_election.cc:290] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.080404 2656 tablet.cc:1721] T 3e94ea76507f4646909ac0c1b2c3136a P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:14.083269 2669 raft_consensus.cc:491] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.083323 1168 tablet_bootstrap.cc:492] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:14.085342 1168 tablet_bootstrap.cc:654] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.092633 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 5265d5eeac83481ca97801e3475886da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.083484 2669 raft_consensus.cc:513] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.095976 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df0250d08214a9783c392071126ed4a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:14.096475 848 leader_election.cc:343] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:14.100261 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8bfb65359dea4a2e810cb2827a3161a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.101945 1164 tablet_bootstrap.cc:492] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:14.102059 1168 tablet_bootstrap.cc:492] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:14.102198 1164 ts_tablet_manager.cc:1397] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.122s user 0.001s sys 0.004s
I20250115 14:52:14.102244 1168 ts_tablet_manager.cc:1397] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.008s
I20250115 14:52:14.103027 1168 raft_consensus.cc:357] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.103293 1168 raft_consensus.cc:383] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.103396 1168 raft_consensus.cc:738] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.103709 1168 consensus_queue.cc:260] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.104925 1168 ts_tablet_manager.cc:1428] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:14.106305 1168 tablet_bootstrap.cc:492] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:14.105088 2655 tablet.cc:1721] T 8c7e70288eb248f1a71646c66e376a63 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:14.109733 2669 leader_election.cc:290] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.109834 1168 tablet_bootstrap.cc:654] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.110015 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df0250d08214a9783c392071126ed4a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:14.110282 1049 raft_consensus.cc:2463] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:14.112020 1162 tablet_bootstrap.cc:492] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.110646 847 leader_election.cc:304] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.112138 1162 ts_tablet_manager.cc:1397] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.003s
I20250115 14:52:14.112963 1162 raft_consensus.cc:357] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.113133 1162 raft_consensus.cc:383] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.113175 1162 raft_consensus.cc:738] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.113396 1162 consensus_queue.cc:260] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.114785 2656 tablet.cc:1721] T 8c7e70288eb248f1a71646c66e376a63 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:14.115437 2669 raft_consensus.cc:2798] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:14.115557 2669 raft_consensus.cc:491] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:14.115661 2669 raft_consensus.cc:3054] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.117008 1162 ts_tablet_manager.cc:1428] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250115 14:52:14.117266 1162 tablet_bootstrap.cc:492] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.118448 2273 raft_consensus.cc:491] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.118561 2273 raft_consensus.cc:513] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.119115 2273 leader_election.cc:290] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.119176 2669 raft_consensus.cc:513] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.119453 2403 raft_consensus.cc:491] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.119589 2403 raft_consensus.cc:513] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.119644 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a79f880415a4b16aabdbb7cd989efdb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:14.120136 983 leader_election.cc:343] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.120187 2403 leader_election.cc:290] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.120610 2273 raft_consensus.cc:491] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.120711 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a79f880415a4b16aabdbb7cd989efdb" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.120723 2273 raft_consensus.cc:513] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:14.121214 984 leader_election.cc:343] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.121316 984 leader_election.cc:304] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.121361 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df0250d08214a9783c392071126ed4a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:14.121501 1048 raft_consensus.cc:3054] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.121730 2563 raft_consensus.cc:2743] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.122092 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f541e916d264738842cd965c7a2a5ec" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.111238 1167 ts_tablet_manager.cc:1918] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.122455 1167 log.cc:1198] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ca128bf3d6fc430298da932a78f54184
W20250115 14:52:14.122535 984 leader_election.cc:343] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.122757 1167 ts_tablet_manager.cc:1939] T ca128bf3d6fc430298da932a78f54184 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:14.123035 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1162) took 1024 ms. Trace:
I20250115 14:52:14.123103 1167 rpcz_store.cc:276] 0115 14:52:13.098305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.099786 (+ 1481us) service_pool.cc:225] Handling call
0115 14:52:13.995934 (+896148us) ts_tablet_manager.cc:1145] Deleting tablet ca128bf3d6fc430298da932a78f54184
0115 14:52:13.995943 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.108448 (+112505us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.111156 (+ 2708us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.123022 (+ 11866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":895996}
I20250115 14:52:14.123284 1168 tablet_bootstrap.cc:492] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:14.123387 1168 ts_tablet_manager.cc:1397] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250115 14:52:14.123401 1167 tablet_replica.cc:331] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:14.123566 1167 raft_consensus.cc:2238] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.123685 1167 raft_consensus.cc:2267] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.123821 1048 raft_consensus.cc:2463] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:14.124116 1168 raft_consensus.cc:357] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.124305 1168 raft_consensus.cc:383] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.124349 1168 raft_consensus.cc:738] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.124522 1168 consensus_queue.cc:260] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.124634 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc363f70f3f44d309f69fac9356c6257" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.125121 1168 ts_tablet_manager.cc:1428] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:14.125341 1168 tablet_bootstrap.cc:492] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:14.125221 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5c321ec5b2c84f01a09fa02678c31b4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.125788 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c321ec5b2c84f01a09fa02678c31b4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.126132 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df0250d08214a9783c392071126ed4a" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:14.126251 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet ca128bf3d6fc430298da932a78f54184 (table default.Table_8f53bbc08e5340aab38bb0ce8ef7aa16 [id=6cdcb9c1b34642d8a792807746ac26f5]) successfully deleted
I20250115 14:52:14.126405 2669 leader_election.cc:290] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.124132 1167 ts_tablet_manager.cc:1905] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:14.126711 848 leader_election.cc:343] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.127921 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_37389ea3789646978b1da3e473f46631"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.128475 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37389ea3789646978b1da3e473f46631 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.128696 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc363f70f3f44d309f69fac9356c6257" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:14.128932 1168 tablet_bootstrap.cc:654] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.129794 1751 ts_tablet_manager.cc:1918] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:14.129884 1751 log.cc:1198] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/97b599c6b8174959b8b66e90e9242057
I20250115 14:52:14.130784 1751 ts_tablet_manager.cc:1939] T 97b599c6b8174959b8b66e90e9242057 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:14.131485 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1152) took 1228 ms. Trace:
I20250115 14:52:14.131567 1751 rpcz_store.cc:276] 0115 14:52:12.902668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:12.949736 (+ 47068us) service_pool.cc:225] Handling call
0115 14:52:13.966841 (+1017105us) ts_tablet_manager.cc:1145] Deleting tablet 97b599c6b8174959b8b66e90e9242057
0115 14:52:13.966854 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:13.987948 (+ 21094us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.129691 (+141743us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.131466 (+ 1775us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1016611}
I20250115 14:52:14.131637 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f541e916d264738842cd965c7a2a5ec" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.131853 1751 tablet_replica.cc:331] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:14.132031 1751 raft_consensus.cc:2238] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.132164 1751 raft_consensus.cc:2267] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:14.132452 983 leader_election.cc:343] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.132547 983 leader_election.cc:304] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.132925 2469 raft_consensus.cc:2743] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.133428 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 97b599c6b8174959b8b66e90e9242057 (table default.Table_e3e5361d9cb144999d02ff4df3e1999b [id=5063a3c3ef3a4b1794a2c8532ab7adf6]) successfully deleted
I20250115 14:52:14.133812 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4e5f5823275b40959b3058fbf28be96d" } new_table_name: "default.Table_288d46f4b2dd44b3a5a4ab52188add56" modify_external_catalogs: true
I20250115 14:52:14.103036 1164 raft_consensus.cc:357] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.134429 1164 raft_consensus.cc:383] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.134497 1164 raft_consensus.cc:738] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.121228 2273 leader_election.cc:290] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.135637 847 leader_election.cc:304] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.135908 2669 raft_consensus.cc:2798] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:14.137365 710 leader_election.cc:304] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:14.137787 2403 raft_consensus.cc:2743] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.137967 2669 raft_consensus.cc:695] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:14.138007 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c5f3bc4addaa42d7acaf157e65707baa" } modify_external_catalogs: true
I20250115 14:52:14.138154 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c5f3bc4addaa42d7acaf157e65707baa" } modify_external_catalogs: true
I20250115 14:52:14.138221 2669 consensus_queue.cc:237] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.138686 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_32ddc413a4ee4d3fb9f1d439fe5cfc3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.139210 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32ddc413a4ee4d3fb9f1d439fe5cfc3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.140657 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d384734a041a4ebf963744b6f24a2572"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.141111 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d384734a041a4ebf963744b6f24a2572 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.141538 1164 consensus_queue.cc:260] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.145434 1164 ts_tablet_manager.cc:1428] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.043s user 0.000s sys 0.001s
I20250115 14:52:14.147527 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_05f540a8c52d40b1b3240deb036757ad" } modify_external_catalogs: true
I20250115 14:52:14.147704 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_05f540a8c52d40b1b3240deb036757ad" } modify_external_catalogs: true
I20250115 14:52:14.149874 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e2b33135a54f436a95b73298d2b17a9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.150490 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2b33135a54f436a95b73298d2b17a9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.162073 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 65a6f83730f74d13823bd15d93538df7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.162334 1166 tablet_replica.cc:331] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:14.162509 1166 raft_consensus.cc:2238] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.162645 1166 raft_consensus.cc:2267] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.163192 1166 ts_tablet_manager.cc:1905] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.168991 1166 ts_tablet_manager.cc:1918] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.169106 1166 log.cc:1198] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/65a6f83730f74d13823bd15d93538df7
I20250115 14:52:14.169385 1166 ts_tablet_manager.cc:1939] T 65a6f83730f74d13823bd15d93538df7 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:14.171478 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 65a6f83730f74d13823bd15d93538df7 (table default.Table_8a0954ff11744933ba4752116830cdd7 [id=a3bf28673dac4ddab579709d6fd8885d]) successfully deleted
I20250115 14:52:14.174090 1167 ts_tablet_manager.cc:1918] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.174278 1167 log.cc:1198] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/6e616d78933e4f84b9ec2acc51450b18
I20250115 14:52:14.174639 1167 ts_tablet_manager.cc:1939] T 6e616d78933e4f84b9ec2acc51450b18 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:14.174960 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1161) took 1100 ms. Trace:
I20250115 14:52:14.177561 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f72ab5996cfd4471996f0b1034d3e422" } new_table_name: "default.Table_736cec83be284849af7e07aa9d729965" modify_external_catalogs: true
I20250115 14:52:14.177623 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b2a9e7fffbe84c459c9b07db2c59dca0" } new_table_name: "default.Table_dcfcf254629a4b96b40ba347ff0dab09" modify_external_catalogs: true
W20250115 14:52:14.178822 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06aa85a23e9e435d9d0d91e807b47a81 has no consensus state
I20250115 14:52:14.179109 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_132293624dd74f678fbc0f2204143d28" } modify_external_catalogs: true
I20250115 14:52:14.179081 1167 rpcz_store.cc:276] 0115 14:52:13.074734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.101857 (+ 27123us) service_pool.cc:225] Handling call
0115 14:52:14.123360 (+1021503us) ts_tablet_manager.cc:1145] Deleting tablet 6e616d78933e4f84b9ec2acc51450b18
0115 14:52:14.123369 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.171126 (+ 47757us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.173999 (+ 2873us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.174945 (+ 946us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1021347}
I20250115 14:52:14.179253 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_132293624dd74f678fbc0f2204143d28" } modify_external_catalogs: true
I20250115 14:52:14.179473 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 6e616d78933e4f84b9ec2acc51450b18 (table default.Table_ff33645127a443efb2b776d808942dba [id=f8c4ca50220a4dbf9b9480213d836e5f]) successfully deleted
I20250115 14:52:14.179958 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1159) took 1105 ms. Trace:
I20250115 14:52:14.180037 1167 rpcz_store.cc:276] 0115 14:52:13.074582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.075184 (+ 602us) service_pool.cc:225] Handling call
0115 14:52:14.179821 (+1104637us) ts_tablet_manager.cc:1145] Deleting tablet 2a1a69b6de5d433f82579b1259548fd2
0115 14:52:14.179834 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.179938 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1071206}
W20250115 14:52:14.180393 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2a1a69b6de5d433f82579b1259548fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1a69b6de5d433f82579b1259548fd2 already in progress: creating tablet
I20250115 14:52:14.180716 1167 tablet_replica.cc:331] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:14.180985 1167 raft_consensus.cc:2238] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.181190 1167 raft_consensus.cc:2267] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.181730 1167 ts_tablet_manager.cc:1905] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.185003 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_11d56e79fc964adbb45d31451183c7d9" } new_table_name: "default.Table_e3a17fa2f348459e90b24f586229b7e1" modify_external_catalogs: true
I20250115 14:52:14.187453 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_633028cf8b914cac8836453b1e242fdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.187992 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_633028cf8b914cac8836453b1e242fdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.189198 2273 raft_consensus.cc:491] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.189345 2273 raft_consensus.cc:513] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.189996 2273 leader_election.cc:290] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.190395 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3280e8ba6db4446aa3e3bfe2836abfa" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:14.190928 984 leader_election.cc:343] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.193212 1164 tablet_bootstrap.cc:492] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:14.190518 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3280e8ba6db4446aa3e3bfe2836abfa" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:14.201572 983 leader_election.cc:343] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.201694 983 leader_election.cc:304] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.202126 2273 raft_consensus.cc:2743] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.190236 1167 ts_tablet_manager.cc:1918] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.202831 1167 log.cc:1198] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/005a4cd0e9f14e76b01afa49d946ee32
I20250115 14:52:14.203033 1162 tablet_bootstrap.cc:654] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.203395 1167 ts_tablet_manager.cc:1939] T 005a4cd0e9f14e76b01afa49d946ee32 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:14.204129 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1160) took 1129 ms. Trace:
I20250115 14:52:14.204306 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_043c9ff0bc444163a82cfc7c4d02c35f" } new_table_name: "default.Table_d515ef8d402847849a21da09dce83509" modify_external_catalogs: true
I20250115 14:52:14.204614 1167 rpcz_store.cc:276] 0115 14:52:13.074657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.075328 (+ 671us) service_pool.cc:225] Handling call
0115 14:52:14.180671 (+1105343us) ts_tablet_manager.cc:1145] Deleting tablet 005a4cd0e9f14e76b01afa49d946ee32
0115 14:52:14.180680 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.187484 (+ 6804us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.190152 (+ 2668us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.204114 (+ 13962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1072002}
I20250115 14:52:14.205330 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 005a4cd0e9f14e76b01afa49d946ee32 (table default.Table_d611f6594f554cfe99c5aa5f2f7ea0e2 [id=cfab75a29f6c4e388b90e1814fa72fae]) successfully deleted
I20250115 14:52:14.205750 1167 tablet_replica.cc:331] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:14.206030 1167 raft_consensus.cc:2238] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.206244 1167 raft_consensus.cc:2267] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.206823 1167 ts_tablet_manager.cc:1905] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.208644 1751 ts_tablet_manager.cc:1905] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.208668 2669 raft_consensus.cc:491] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.209426 2669 raft_consensus.cc:513] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:14.210915 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 5ed0b0ccdcff423d9bb2b9a83511616d (table default.Table_72e496f5859b463caaac692c42725308 [id=c25247739e93462ca2f16a3076a614cb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5ed0b0ccdcff423d9bb2b9a83511616d consensus state has no leader
I20250115 14:52:14.211946 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9f0ecfceecf4acdbf2c2fbec88ff3cb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.212311 2669 leader_election.cc:290] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:14.212669 848 leader_election.cc:343] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.214926 1751 ts_tablet_manager.cc:1918] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.215041 1751 log.cc:1198] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/48b15de9bf8943cdb9e5abd943484993
I20250115 14:52:14.215337 1751 ts_tablet_manager.cc:1939] T 48b15de9bf8943cdb9e5abd943484993 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:14.215626 847 leader_election.cc:343] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: a9f0ecfceecf4acdbf2c2fbec88ff3cb
I20250115 14:52:14.215725 847 leader_election.cc:304] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.216075 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1167) took 1202 ms. Trace:
I20250115 14:52:14.216200 2669 raft_consensus.cc:2743] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.217089 1168 tablet_bootstrap.cc:492] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:14.217216 1168 ts_tablet_manager.cc:1397] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.092s user 0.010s sys 0.007s
I20250115 14:52:14.217933 1168 raft_consensus.cc:357] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.218106 1168 raft_consensus.cc:383] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.218150 1168 raft_consensus.cc:738] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.218336 1168 consensus_queue.cc:260] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.218835 1168 ts_tablet_manager.cc:1428] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:14.219107 1168 tablet_bootstrap.cc:492] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:14.232664 1162 tablet_bootstrap.cc:492] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.232831 1162 ts_tablet_manager.cc:1397] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.116s user 0.001s sys 0.002s
I20250115 14:52:14.233538 1162 raft_consensus.cc:357] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.233733 1162 raft_consensus.cc:383] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.233853 1162 raft_consensus.cc:738] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.234324 1162 consensus_queue.cc:260] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.238111 1162 ts_tablet_manager.cc:1428] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250115 14:52:14.240190 1162 tablet_bootstrap.cc:492] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.216161 1751 rpcz_store.cc:276] 0115 14:52:13.013492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.013537 (+ 45us) service_pool.cc:225] Handling call
0115 14:52:14.131806 (+1118269us) ts_tablet_manager.cc:1145] Deleting tablet 48b15de9bf8943cdb9e5abd943484993
0115 14:52:14.131815 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.211989 (+ 80174us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.214830 (+ 2841us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.216057 (+ 1227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118115}
I20250115 14:52:14.251667 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 48b15de9bf8943cdb9e5abd943484993 (table default.Table_a7e733ef2d844d238685b7d25b53c151 [id=c40fb10d4df34d58a790b3d4e39b48e5]) successfully deleted
I20250115 14:52:14.252456 1164 tablet_bootstrap.cc:654] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.257844 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_2ea7cdeffde54825be02af36704833a0 [id=13d5cd4ccf2245319f40495141007b58] alter complete (version 1)
W20250115 14:52:14.257928 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet daacfd23ec90436089382270e45f9e9b consensus state has no leader
I20250115 14:52:14.258620 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_597d1c61c9bc4e398eaf813ad56c8841"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.259135 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_597d1c61c9bc4e398eaf813ad56c8841 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.259282 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b28114e5556947c4aaba883427f90c2a" } modify_external_catalogs: true
I20250115 14:52:14.259434 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b28114e5556947c4aaba883427f90c2a" } modify_external_catalogs: true
W20250115 14:52:14.260579 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7db35ff0383549de86230bb1eb89389f consensus state has no leader
I20250115 14:52:14.260975 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a829fe34660949f4808429fc815c1c11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.261399 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a829fe34660949f4808429fc815c1c11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:14.262495 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d13576f09c8f4fcd906b696b4811b48a consensus state has no leader
I20250115 14:52:14.262780 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_47fb50f420d34a0f946869232d283ecc" } modify_external_catalogs: true
I20250115 14:52:14.262907 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_47fb50f420d34a0f946869232d283ecc" } modify_external_catalogs: true
I20250115 14:52:14.263357 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet bfc6e5c81a6d40a3a2054450d0478ad6
I20250115 14:52:14.263972 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0b03e1670f0c48dab3a8f103e6bf6dbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.264408 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b03e1670f0c48dab3a8f103e6bf6dbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.265111 1164 tablet_bootstrap.cc:492] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:14.265249 1164 ts_tablet_manager.cc:1397] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.119s user 0.001s sys 0.001s
I20250115 14:52:14.265686 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 288b0e455a854464940f65a2ab909d0b from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:14.265933 1164 raft_consensus.cc:357] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.266112 1164 raft_consensus.cc:383] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.266165 1164 raft_consensus.cc:738] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.266358 1164 consensus_queue.cc:260] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.266830 1164 ts_tablet_manager.cc:1428] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:14.267131 1164 tablet_bootstrap.cc:492] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:14.267506 738 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc6e5c81a6d40a3a2054450d0478ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.268080 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc6e5c81a6d40a3a2054450d0478ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.269551 1164 tablet_bootstrap.cc:654] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:14.270035 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet bfc6e5c81a6d40a3a2054450d0478ad6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfc6e5c81a6d40a3a2054450d0478ad6 already in progress: creating tablet
I20250115 14:52:14.270716 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet f3280e8ba6db4446aa3e3bfe2836abfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.270956 882 tablet_service.cc:1514] Processing DeleteTablet for tablet bfc6e5c81a6d40a3a2054450d0478ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.273200 1168 tablet_bootstrap.cc:654] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.279013 738 tablet_service.cc:1514] Processing DeleteTablet for tablet b803d8e103a5423f85c75c9006809b17 with delete_type TABLET_DATA_DELETED (replaced by tablet 6437873733584722979c1e9ae8b0730b) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.279242 1166 tablet_replica.cc:331] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:14.279408 1166 raft_consensus.cc:2238] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.279551 1166 raft_consensus.cc:2267] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.280090 1166 ts_tablet_manager.cc:1905] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.282711 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 36de948083a946eba1a96f276e4f4f91 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e33ebec8b64f2398a7f1003b15cee7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.286422 1167 ts_tablet_manager.cc:1918] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.286661 1166 ts_tablet_manager.cc:1918] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.286751 1166 log.cc:1198] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b803d8e103a5423f85c75c9006809b17
I20250115 14:52:14.286988 1166 ts_tablet_manager.cc:1939] T b803d8e103a5423f85c75c9006809b17 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:14.288805 1167 log.cc:1198] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/a2fe51b51994491ab38a644bfc55e3ac
I20250115 14:52:14.289351 1167 ts_tablet_manager.cc:1939] T a2fe51b51994491ab38a644bfc55e3ac P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:14.289829 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b803d8e103a5423f85c75c9006809b17 (table default.Table_4dff2e869f8141e09dd757d6e47118f4 [id=43852683a5a64e1b85403395fa3d210c]) successfully deleted
I20250115 14:52:14.290241 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1166) took 1060 ms. Trace:
I20250115 14:52:14.290571 1167 rpcz_store.cc:276] 0115 14:52:13.230261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.231113 (+ 852us) service_pool.cc:225] Handling call
0115 14:52:14.205701 (+974588us) ts_tablet_manager.cc:1145] Deleting tablet a2fe51b51994491ab38a644bfc55e3ac
0115 14:52:14.205711 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.283621 (+ 77910us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.286341 (+ 2720us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.290224 (+ 3883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":974394}
I20250115 14:52:14.291296 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet a2fe51b51994491ab38a644bfc55e3ac (table default.Table_9f7b9d2bd72741fbb3132278c8a77d1c [id=15f2a471b89449459d7dbf804b4e922c]) successfully deleted
I20250115 14:52:14.291826 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1169) took 1001 ms. Trace:
I20250115 14:52:14.292325 1167 rpcz_store.cc:276] 0115 14:52:13.290260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.290305 (+ 45us) service_pool.cc:225] Handling call
0115 14:52:14.291630 (+1001325us) ts_tablet_manager.cc:1145] Deleting tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c
0115 14:52:14.291641 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.291752 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":997538}
W20250115 14:52:14.293081 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c already in progress: creating tablet
W20250115 14:52:14.293829 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0936fa510a454b8e91d6ef2b345b18df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0936fa510a454b8e91d6ef2b345b18df already in progress: creating tablet
I20250115 14:52:14.294473 1167 tablet_replica.cc:331] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:14.294873 1167 raft_consensus.cc:2238] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:14.295439 1167 raft_consensus.cc:2267] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.296155 2273 raft_consensus.cc:491] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.296315 2273 raft_consensus.cc:513] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.296420 1167 ts_tablet_manager.cc:1905] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.296921 2273 leader_election.cc:290] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.254369 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1169) took 1193 ms. Trace:
I20250115 14:52:14.300441 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed777d7496514ec5986a7e6f53d532b1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.300390 1751 rpcz_store.cc:276] 0115 14:52:13.061037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.061084 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:14.252625 (+1191541us) ts_tablet_manager.cc:1145] Deleting tablet 1fc659e2d2d943a99aef530e7d71df88
0115 14:52:14.252636 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.254034 (+ 1398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191052}
W20250115 14:52:14.300839 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 1fc659e2d2d943a99aef530e7d71df88 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fc659e2d2d943a99aef530e7d71df88 already in progress: creating tablet
I20250115 14:52:14.301113 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed777d7496514ec5986a7e6f53d532b1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.301267 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1168) took 1240 ms. Trace:
I20250115 14:52:14.301334 1751 rpcz_store.cc:276] 0115 14:52:13.060834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.068016 (+ 7182us) service_pool.cc:225] Handling call
0115 14:52:14.301150 (+1233134us) ts_tablet_manager.cc:1145] Deleting tablet 2a1a69b6de5d433f82579b1259548fd2
0115 14:52:14.301160 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.301250 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1203024}
W20250115 14:52:14.301683 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2a1a69b6de5d433f82579b1259548fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a1a69b6de5d433f82579b1259548fd2 already in progress: creating tablet
W20250115 14:52:14.302058 984 leader_election.cc:343] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.302121 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1172) took 1147 ms. Trace:
I20250115 14:52:14.302173 1751 rpcz_store.cc:276] 0115 14:52:13.154582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.154637 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:14.302005 (+1147368us) ts_tablet_manager.cc:1145] Deleting tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c
0115 14:52:14.302014 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.302107 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1147180}
W20250115 14:52:14.302469 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c already in progress: creating tablet
W20250115 14:52:14.302470 983 leader_election.cc:343] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.302556 983 leader_election.cc:304] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.302790 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1183) took 1054 ms. Trace:
I20250115 14:52:14.302851 1751 rpcz_store.cc:276] 0115 14:52:13.248079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.248129 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:14.302669 (+1054540us) ts_tablet_manager.cc:1145] Deleting tablet 2a005ad11e7941fab96b5565d55cfaf7
0115 14:52:14.302678 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.302776 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054318}
I20250115 14:52:14.302923 2273 raft_consensus.cc:2743] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:14.303174 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2a005ad11e7941fab96b5565d55cfaf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a005ad11e7941fab96b5565d55cfaf7 already in progress: creating tablet
I20250115 14:52:14.303449 1751 tablet_replica.cc:331] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:14.303764 1751 raft_consensus.cc:2238] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.303985 1751 raft_consensus.cc:2267] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.304572 1751 ts_tablet_manager.cc:1905] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.311548 2669 raft_consensus.cc:491] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.311676 2669 raft_consensus.cc:513] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.312467 1162 tablet_bootstrap.cc:654] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.315340 2657 raft_consensus.cc:491] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.315459 2657 raft_consensus.cc:513] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.316094 2657 leader_election.cc:290] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.316844 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325cd3fd8b4d447998822163e575e758" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:14.317353 848 leader_election.cc:343] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.319057 1168 tablet_bootstrap.cc:492] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
W20250115 14:52:14.319092 847 leader_election.cc:343] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 325cd3fd8b4d447998822163e575e758
I20250115 14:52:14.319183 1168 ts_tablet_manager.cc:1397] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.002s
I20250115 14:52:14.319172 847 leader_election.cc:304] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.319711 2657 raft_consensus.cc:2743] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.319931 1168 raft_consensus.cc:357] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.320099 1168 raft_consensus.cc:383] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.320149 1168 raft_consensus.cc:738] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.320333 1168 consensus_queue.cc:260] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.320991 1168 ts_tablet_manager.cc:1428] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:14.324461 1167 ts_tablet_manager.cc:1918] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:14.326011 1168 tablet_bootstrap.cc:492] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:14.330071 1162 tablet_bootstrap.cc:492] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.330209 1162 ts_tablet_manager.cc:1397] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.000s
I20250115 14:52:14.330492 1168 tablet_bootstrap.cc:654] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.330832 1162 raft_consensus.cc:357] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.331027 1162 raft_consensus.cc:383] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.331073 1162 raft_consensus.cc:738] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.331264 1162 consensus_queue.cc:260] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.331815 1162 ts_tablet_manager.cc:1428] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:14.332122 1162 tablet_bootstrap.cc:492] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.333686 1162 tablet_bootstrap.cc:654] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.333698 1167 log.cc:1198] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d5ffa997ca6c46559830b1dedf2d1a66
I20250115 14:52:14.334342 1167 ts_tablet_manager.cc:1939] T d5ffa997ca6c46559830b1dedf2d1a66 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:14.341380 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet d1259256b7744e739581125611abbe67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.350281 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]) successfully deleted
I20250115 14:52:14.351073 1751 ts_tablet_manager.cc:1918] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:14.351181 1751 log.cc:1198] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d5ffa997ca6c46559830b1dedf2d1a66
I20250115 14:52:14.351532 1751 ts_tablet_manager.cc:1939] T d5ffa997ca6c46559830b1dedf2d1a66 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:14.351567 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2a005ad11e7941fab96b5565d55cfaf7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a005ad11e7941fab96b5565d55cfaf7 already in progress: creating tablet
W20250115 14:52:14.355580 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 58555us
W20250115 14:52:14.356314 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e6821f4c34f042c8b4819176fe4418ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6821f4c34f042c8b4819176fe4418ff already in progress: creating tablet
W20250115 14:52:14.356974 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f1e3879802464acaa279dfe4fbb634bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e3879802464acaa279dfe4fbb634bb already in progress: creating tablet
W20250115 14:52:14.357921 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 already in progress: creating tablet
I20250115 14:52:14.358091 1168 tablet_bootstrap.cc:492] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:14.358260 1168 ts_tablet_manager.cc:1397] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.007s
W20250115 14:52:14.358459 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c497fb29752d4ed7bda207301f90cf05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c497fb29752d4ed7bda207301f90cf05 already in progress: creating tablet
W20250115 14:52:14.358964 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 23318b1ad0774a76880def17a64ac059 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23318b1ad0774a76880def17a64ac059 already in progress: creating tablet
I20250115 14:52:14.359194 1168 raft_consensus.cc:357] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:14.359474 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 356a0dfd7bb746548e0fb5b6a0935318 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 356a0dfd7bb746548e0fb5b6a0935318 already in progress: creating tablet
W20250115 14:52:14.359993 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 0f82d944a7e04ace8e4addb3897d2c8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f82d944a7e04ace8e4addb3897d2c8e already in progress: creating tablet
I20250115 14:52:14.360153 1168 raft_consensus.cc:383] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.360348 1168 raft_consensus.cc:738] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.360705 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_46ab5287629b4006b72bb93ccde2b813"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.361269 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46ab5287629b4006b72bb93ccde2b813 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.361316 1168 consensus_queue.cc:260] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:14.363610 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 50f8d93d655a447cb9e92997342a61a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50f8d93d655a447cb9e92997342a61a7 already in progress: creating tablet
W20250115 14:52:14.364562 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 93aaea90b0924327807e7f594154b14e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93aaea90b0924327807e7f594154b14e already in progress: creating tablet
I20250115 14:52:14.369632 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_adabfc70f8eb4826ae67c16382a57ee9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.370147 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adabfc70f8eb4826ae67c16382a57ee9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.372948 1164 tablet_bootstrap.cc:492] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:14.373023 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 13a4837b72eb4008ad612a9fc86dfac7 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:14.373541 1164 ts_tablet_manager.cc:1397] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.106s user 0.013s sys 0.016s
I20250115 14:52:14.375385 1164 raft_consensus.cc:357] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.375949 1164 raft_consensus.cc:383] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.376109 1164 raft_consensus.cc:738] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.376402 1167 tablet_replica.cc:331] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:14.376434 1164 consensus_queue.cc:260] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.376953 1167 raft_consensus.cc:2238] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.377406 1167 raft_consensus.cc:2267] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.378120 1167 ts_tablet_manager.cc:1905] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.382463 1164 ts_tablet_manager.cc:1428] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250115 14:52:14.382918 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 0651be54e0b244a4bace9f47efc9b58a with delete_type TABLET_DATA_DELETED (replaced by tablet 324c810773f249db9e8325b09805141a) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.383721 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5265d5eeac83481ca97801e3475886da" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:14.384267 848 leader_election.cc:343] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.384388 1166 tablet_replica.cc:331] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:14.384567 1166 raft_consensus.cc:2238] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.384696 1166 raft_consensus.cc:2267] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:14.385759 847 leader_election.cc:343] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 5265d5eeac83481ca97801e3475886da
I20250115 14:52:14.385866 847 leader_election.cc:304] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.386101 2657 raft_consensus.cc:2743] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.386878 1164 tablet_bootstrap.cc:492] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:14.386996 2669 leader_election.cc:290] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.388142 1166 ts_tablet_manager.cc:1905] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.361080 1162 tablet_bootstrap.cc:492] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.389514 1162 ts_tablet_manager.cc:1397] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20250115 14:52:14.390245 1162 raft_consensus.cc:357] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.390430 1162 raft_consensus.cc:383] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.390481 1162 raft_consensus.cc:738] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.390712 1162 consensus_queue.cc:260] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.391595 1162 ts_tablet_manager.cc:1428] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:14.352274 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1184) took 1090 ms. Trace:
I20250115 14:52:14.391736 1751 rpcz_store.cc:276] 0115 14:52:13.262261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.263179 (+ 918us) service_pool.cc:225] Handling call
0115 14:52:14.303406 (+1040227us) ts_tablet_manager.cc:1145] Deleting tablet d5ffa997ca6c46559830b1dedf2d1a66
0115 14:52:14.303415 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.307497 (+ 4082us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.350958 (+ 43461us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.352256 (+ 1298us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1040050}
I20250115 14:52:14.392130 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d5ffa997ca6c46559830b1dedf2d1a66 (table default.Table_cac9e05c8cae4cab9a55c08b3dc25648 [id=2c2d7017c98f49d09e57327f21412e91]) successfully deleted
I20250115 14:52:14.392529 1162 tablet_bootstrap.cc:492] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.394044 1166 ts_tablet_manager.cc:1918] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.394140 1166 log.cc:1198] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/0651be54e0b244a4bace9f47efc9b58a
I20250115 14:52:14.394136 1162 tablet_bootstrap.cc:654] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.387538 1168 ts_tablet_manager.cc:1428] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20250115 14:52:14.395377 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1185) took 1127 ms. Trace:
I20250115 14:52:14.395455 1751 rpcz_store.cc:276] 0115 14:52:13.267739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.267797 (+ 58us) service_pool.cc:225] Handling call
0115 14:52:14.395260 (+1127463us) ts_tablet_manager.cc:1145] Deleting tablet 0936fa510a454b8e91d6ef2b345b18df
0115 14:52:14.395270 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.395359 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127288}
W20250115 14:52:14.395866 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0936fa510a454b8e91d6ef2b345b18df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0936fa510a454b8e91d6ef2b345b18df already in progress: creating tablet
I20250115 14:52:14.397222 1168 tablet_bootstrap.cc:492] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:14.397889 1166 ts_tablet_manager.cc:1939] T 0651be54e0b244a4bace9f47efc9b58a P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:14.398548 1751 tablet_replica.cc:331] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:14.398720 1751 raft_consensus.cc:2238] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.398851 1751 raft_consensus.cc:2267] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.398911 2655 tablet.cc:1721] T ea8ab6a34caa486ab929af2cac3ce957 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:14.399324 1751 ts_tablet_manager.cc:1905] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.400069 2660 tablet.cc:1721] T ea8ab6a34caa486ab929af2cac3ce957 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:14.400589 1167 ts_tablet_manager.cc:1918] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.401862 1164 tablet_bootstrap.cc:654] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.408200 1167 log.cc:1198] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/a9f0ecfceecf4acdbf2c2fbec88ff3cb
I20250115 14:52:14.408584 1167 ts_tablet_manager.cc:1939] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:14.409514 2710 tablet.cc:1721] T ea8ab6a34caa486ab929af2cac3ce957 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:14.409802 1167 tablet_replica.cc:331] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:14.409971 1167 raft_consensus.cc:2238] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.410092 1167 raft_consensus.cc:2267] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.410573 1167 ts_tablet_manager.cc:1905] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.412315 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb (table default.Table_1c51a5aaced84915a1dba864731460fe [id=ec452771986240f4a7654505a30d3d9d]) successfully deleted
W20250115 14:52:14.412606 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ec9e7c8e59aa4ea59bad35c8a7591a79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec9e7c8e59aa4ea59bad35c8a7591a79 already in progress: creating tablet
I20250115 14:52:14.403596 1168 tablet_bootstrap.cc:654] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:14.412830 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7cf17435b88c4e87800daefcd4fa35ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cf17435b88c4e87800daefcd4fa35ec already in progress: creating tablet
I20250115 14:52:14.413113 1162 tablet_bootstrap.cc:492] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.413236 1162 ts_tablet_manager.cc:1397] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20250115 14:52:14.413085 2273 raft_consensus.cc:491] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.413511 2273 raft_consensus.cc:513] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.413908 1162 raft_consensus.cc:357] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.414076 1162 raft_consensus.cc:383] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.414096 2273 leader_election.cc:290] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.414127 1162 raft_consensus.cc:738] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.414286 1162 consensus_queue.cc:260] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.414551 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f2bc62621d402cacfcbe2eb74173f9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.414744 1162 ts_tablet_manager.cc:1428] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:14.415004 1162 tablet_bootstrap.cc:492] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:14.415133 983 leader_election.cc:343] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.415639 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f2bc62621d402cacfcbe2eb74173f9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:14.416177 984 leader_election.cc:343] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.416270 984 leader_election.cc:304] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.416875 2273 raft_consensus.cc:2743] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.421994 1167 ts_tablet_manager.cc:1918] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.425063 1164 tablet_bootstrap.cc:492] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:14.425253 1164 ts_tablet_manager.cc:1397] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.004s
I20250115 14:52:14.425976 1164 raft_consensus.cc:357] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.426226 1164 raft_consensus.cc:383] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.426316 1164 raft_consensus.cc:738] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.426563 1164 consensus_queue.cc:260] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.427219 1162 tablet_bootstrap.cc:654] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.429478 1164 ts_tablet_manager.cc:1428] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250115 14:52:14.429656 1168 tablet_bootstrap.cc:492] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:14.429534 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_da1665493268478cbf1a922a31e0af9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:14.429930 1168 ts_tablet_manager.cc:1397] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.001s
W20250115 14:52:14.430086 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da1665493268478cbf1a922a31e0af9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.430912 1168 raft_consensus.cc:357] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.431082 1168 raft_consensus.cc:383] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.431131 1168 raft_consensus.cc:738] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.431344 1168 consensus_queue.cc:260] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.431854 1168 ts_tablet_manager.cc:1428] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:14.432808 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_65c87128c01649dea82c71db78a50391"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.433293 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65c87128c01649dea82c71db78a50391 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.434461 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7584e090790949b3b7d23a1ced5078be" } new_table_name: "default.Table_baa1a5a968444c6ba2e8094bf8cf3163" modify_external_catalogs: true
I20250115 14:52:14.435211 1164 tablet_bootstrap.cc:492] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:14.435690 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e1c85cd6cf714918b357548b0bc26251"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.436146 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1c85cd6cf714918b357548b0bc26251 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.436808 1164 tablet_bootstrap.cc:654] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.439972 1168 tablet_bootstrap.cc:492] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:14.441437 1168 tablet_bootstrap.cc:654] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.442108 1167 log.cc:1198] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/b76d1bbf90c6416ca60805c9d8053bd4
I20250115 14:52:14.442499 1167 ts_tablet_manager.cc:1939] T b76d1bbf90c6416ca60805c9d8053bd4 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:14.444933 1162 tablet_bootstrap.cc:492] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.445081 1162 ts_tablet_manager.cc:1397] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250115 14:52:14.445755 1162 raft_consensus.cc:357] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.445945 1162 raft_consensus.cc:383] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.446007 1162 raft_consensus.cc:738] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.446439 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 0651be54e0b244a4bace9f47efc9b58a (table default.Table_93023df4f27c40058830c1e90a1b0243 [id=eec67f558ecb4cffaa82516d5e51906d]) successfully deleted
I20250115 14:52:14.448432 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet b76d1bbf90c6416ca60805c9d8053bd4 (table default.Table_9edd581e8fc4457697465360c4504ff1 [id=0ef6f07cf29e46e69bf2a4449187a4e7]) successfully deleted
W20250115 14:52:14.449101 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet abe3c66176664fa99c7a1fb480b56cca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abe3c66176664fa99c7a1fb480b56cca already in progress: creating tablet
I20250115 14:52:14.449417 1167 tablet_replica.cc:331] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:14.449652 1167 raft_consensus.cc:2238] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.449870 1167 raft_consensus.cc:2267] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.450544 1167 ts_tablet_manager.cc:1905] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.457610 448 catalog_manager.cc:5526] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:14.458251 448 catalog_manager.cc:5526] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:14.458580 448 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet ad988e25800f4af9a069b5a80582f749 (table default.Table_162eb94aa1674d21ae644021401a54e8 [id=e3887aa962874cfaa7ee89503ca1a49a]). Expected version 1 got 0
I20250115 14:52:14.459241 1167 ts_tablet_manager.cc:1918] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.463968 1029 tablet_service.cc:1467] Processing CreateTablet for tablet afea664025e54f6099d825c8f7a0ea1a (DEFAULT_TABLE table=default.Table_1033f77f38414857a1255c05f93935d2 [id=b079ea0cfd244f0abda1500173404ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.464421 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afea664025e54f6099d825c8f7a0ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.466565 755 tablet_service.cc:1467] Processing CreateTablet for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e (DEFAULT_TABLE table=default.Table_06ae3b8ba32f4b05af77d455a746ecd2 [id=6ac47a59a5f249d48f4d69f7bb28f02c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.466919 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.468536 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 95f9f9d5c9924479a86b8e2d22e0bf14 (DEFAULT_TABLE table=default.Table_56d30c076381477494fb1b08ef4d4c46 [id=da1ebfcda8a043a9891fd3e0aaa24e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.468885 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f9f9d5c9924479a86b8e2d22e0bf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.472577 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 7b940dce7de14c1b844f8835d436a3d8 (DEFAULT_TABLE table=default.Table_7584e090790949b3b7d23a1ced5078be [id=3872d9b0d70547f79ca6cb24ea506946]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.472931 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b940dce7de14c1b844f8835d436a3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.474995 738 tablet_service.cc:1467] Processing CreateTablet for tablet afea664025e54f6099d825c8f7a0ea1a (DEFAULT_TABLE table=default.Table_1033f77f38414857a1255c05f93935d2 [id=b079ea0cfd244f0abda1500173404ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.475314 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afea664025e54f6099d825c8f7a0ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.475356 1164 tablet_bootstrap.cc:492] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:14.475455 1164 ts_tablet_manager.cc:1397] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.007s
I20250115 14:52:14.476151 1164 raft_consensus.cc:357] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.476354 1164 raft_consensus.cc:383] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.476403 1164 raft_consensus.cc:738] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.476580 1164 consensus_queue.cc:260] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.477015 1164 ts_tablet_manager.cc:1428] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:14.480540 741 tablet_service.cc:1467] Processing CreateTablet for tablet 95f9f9d5c9924479a86b8e2d22e0bf14 (DEFAULT_TABLE table=default.Table_56d30c076381477494fb1b08ef4d4c46 [id=da1ebfcda8a043a9891fd3e0aaa24e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.480901 743 tablet_service.cc:1467] Processing CreateTablet for tablet 7b940dce7de14c1b844f8835d436a3d8 (DEFAULT_TABLE table=default.Table_7584e090790949b3b7d23a1ced5078be [id=3872d9b0d70547f79ca6cb24ea506946]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.481204 753 tablet_service.cc:1467] Processing CreateTablet for tablet 58d1f35f4e1f4c2b8d9516df0586ea44 (DEFAULT_TABLE table=default.Table_554ebf7919a940628afd939485c575d9 [id=e75dfaeb216844d082092fe0d36e10cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.481812 2273 raft_consensus.cc:491] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.481946 2273 raft_consensus.cc:513] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.480880 1164 tablet_bootstrap.cc:492] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:14.482607 2273 leader_election.cc:290] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.483603 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7830c683a109448ab06c15b5de692e8b
I20250115 14:52:14.483649 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 061234cb76b842a7a374ddd5f44fd9a1
I20250115 14:52:14.484198 1164 tablet_bootstrap.cc:654] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.485917 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_84014c06851e40abab7729b00a5aac85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:14.486333 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7a6e5fd036c94dcd843bea04f7377aee" } new_table_name: "default.Table_049f35b1737c479bbff44ef1afe050e2" modify_external_catalogs: true
W20250115 14:52:14.486423 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84014c06851e40abab7729b00a5aac85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.488940 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 58d1f35f4e1f4c2b8d9516df0586ea44 (DEFAULT_TABLE table=default.Table_554ebf7919a940628afd939485c575d9 [id=e75dfaeb216844d082092fe0d36e10cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.489378 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d1f35f4e1f4c2b8d9516df0586ea44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.490945 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d743d5b92e184c949a71cff40336acf9" } new_table_name: "default.Table_b2803f567cf849a480c49e93808b1a19" modify_external_catalogs: true
I20250115 14:52:14.492280 2563 raft_consensus.cc:491] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.492444 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 7830c683a109448ab06c15b5de692e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.492426 2563 raft_consensus.cc:513] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.492630 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 061234cb76b842a7a374ddd5f44fd9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.492756 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f2bc62621d402cacfcbe2eb74173f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.493045 2563 leader_election.cc:290] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.493264 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 7830c683a109448ab06c15b5de692e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.496232 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9ad3cda1056349a789a1751c5d386d0c
I20250115 14:52:14.496817 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_538a75b4e6e54b619c8a011996f4f971" } new_table_name: "default.Table_172dcf8ef25b424ca247c507a87c8d0f" modify_external_catalogs: true
I20250115 14:52:14.497264 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 13a4837b72eb4008ad612a9fc86dfac7
I20250115 14:52:14.497617 455 catalog_manager.cc:6596] ReplaceTablet: tablet 13a4837b72eb4008ad612a9fc86dfac7 deleted and replaced by tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8
I20250115 14:52:14.497865 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_014f98e470c94ba4b3446630153a9e75" } modify_external_catalogs: true
I20250115 14:52:14.497999 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_014f98e470c94ba4b3446630153a9e75" } modify_external_catalogs: true
I20250115 14:52:14.498039 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cc9a98342a67450a92d9f329a3cfc05e" } new_table_name: "default.Table_8d28e0f4d979404cb5c1b592b69dc7f7" modify_external_catalogs: true
I20250115 14:52:14.499293 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 24729156769443f8926577bae7707847
I20250115 14:52:14.499349 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_254b5a8c44db4a788415c8b69471bf9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.499874 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_254b5a8c44db4a788415c8b69471bf9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:14.500113 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b940dce7de14c1b844f8835d436a3d8 consensus state has no leader
I20250115 14:52:14.500564 2273 raft_consensus.cc:491] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.500694 2273 raft_consensus.cc:513] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.501155 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a4837b72eb4008ad612a9fc86dfac7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.501293 2273 leader_election.cc:290] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.501332 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad3cda1056349a789a1751c5d386d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.501448 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 24729156769443f8926577bae7707847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.510032 1015 tablet_service.cc:1467] Processing CreateTablet for tablet e41d8ab30f974bdcba94346460b504a6 (DEFAULT_TABLE table=default.Table_c3d7415d4479485b9629973554670e05 [id=36e4935a4a1f4dc995e9c7bf314e0ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.510427 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41d8ab30f974bdcba94346460b504a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.512343 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 061234cb76b842a7a374ddd5f44fd9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.512571 1024 tablet_service.cc:1467] Processing CreateTablet for tablet d62f1dfe58874fa39def65ea38c791e9 (DEFAULT_TABLE table=default.Table_7c016ac18c7c4026b8e60a53f889bf0d [id=dbaa86e0c6294e8aba837f930d95b144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.512902 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d62f1dfe58874fa39def65ea38c791e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.513473 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 0651be54e0b244a4bace9f47efc9b58a with delete_type TABLET_DATA_DELETED (replaced by tablet 324c810773f249db9e8325b09805141a) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.522766 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 37970a1f9073400ebf973c4ac3510a72 (DEFAULT_TABLE table=default.Table_6c72bc6feb014b768d58cf8c6095fa8a [id=ce6ac893855e4cfb9a57ffd197bb938a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.523154 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37970a1f9073400ebf973c4ac3510a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.523396 447 master_service.cc:946] ReplaceTablet: received request to replace tablet 3e834623f9c7482d965afd23c89a2510 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:14.525471 1022 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc (DEFAULT_TABLE table=default.Table_c49292ffc41542abb7c1104187ce5cba [id=085b32beb8424ec98e18d011609b6629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.525440 1019 tablet_service.cc:1467] Processing CreateTablet for tablet a29da671e4f6496b9eb46cac73a1c32b (DEFAULT_TABLE table=default.Table_b322d270a8f74d2e95bb772545cd9ade [id=0a734bb5c8fa47d099cb23c1f20f2716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.526126 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29da671e4f6496b9eb46cac73a1c32b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.525403 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a0e115eb01814dcbb4f91c369a1c6835 (DEFAULT_TABLE table=default.Table_02dcf1af493a4be5857c31c1c50836a7 [id=e9dbf1dbf63741e2b7f8a051bead5973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.528563 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e115eb01814dcbb4f91c369a1c6835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.528437 1162 consensus_queue.cc:260] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.529022 1162 ts_tablet_manager.cc:1428] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.084s user 0.002s sys 0.000s
I20250115 14:52:14.529268 1162 tablet_bootstrap.cc:492] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.531194 1162 tablet_bootstrap.cc:654] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.525342 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 3efaa97b9c88475db3657f4136869d9c (DEFAULT_TABLE table=default.Table_1d7ea0f078e148088d47b6e1b3b1903b [id=c10b4b90d6134ae392d22d32f832f839]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.532562 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efaa97b9c88475db3657f4136869d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.532694 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad3cda1056349a789a1751c5d386d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.533324 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a4837b72eb4008ad612a9fc86dfac7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.533941 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 24729156769443f8926577bae7707847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.534792 755 tablet_service.cc:1467] Processing CreateTablet for tablet d62f1dfe58874fa39def65ea38c791e9 (DEFAULT_TABLE table=default.Table_7c016ac18c7c4026b8e60a53f889bf0d [id=dbaa86e0c6294e8aba837f930d95b144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.535442 754 tablet_service.cc:1467] Processing CreateTablet for tablet e41d8ab30f974bdcba94346460b504a6 (DEFAULT_TABLE table=default.Table_c3d7415d4479485b9629973554670e05 [id=36e4935a4a1f4dc995e9c7bf314e0ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.535778 756 tablet_service.cc:1467] Processing CreateTablet for tablet 05f9945c40624c3aafda52909edcf467 (DEFAULT_TABLE table=default.Table_c586d1fdfee24db2b1b3e9196fca1641 [id=1d95d1ca67fc4239abd896a78a80fb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.536063 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 05f9945c40624c3aafda52909edcf467 (DEFAULT_TABLE table=default.Table_c586d1fdfee24db2b1b3e9196fca1641 [id=1d95d1ca67fc4239abd896a78a80fb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.536095 750 tablet_service.cc:1467] Processing CreateTablet for tablet a29da671e4f6496b9eb46cac73a1c32b (DEFAULT_TABLE table=default.Table_b322d270a8f74d2e95bb772545cd9ade [id=0a734bb5c8fa47d099cb23c1f20f2716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.536240 748 tablet_service.cc:1467] Processing CreateTablet for tablet 3efaa97b9c88475db3657f4136869d9c (DEFAULT_TABLE table=default.Table_1d7ea0f078e148088d47b6e1b3b1903b [id=c10b4b90d6134ae392d22d32f832f839]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.535778 746 tablet_service.cc:1467] Processing CreateTablet for tablet 37970a1f9073400ebf973c4ac3510a72 (DEFAULT_TABLE table=default.Table_6c72bc6feb014b768d58cf8c6095fa8a [id=ce6ac893855e4cfb9a57ffd197bb938a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.536466 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.536820 744 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc (DEFAULT_TABLE table=default.Table_c49292ffc41542abb7c1104187ce5cba [id=085b32beb8424ec98e18d011609b6629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.537420 737 tablet_service.cc:1467] Processing CreateTablet for tablet a0e115eb01814dcbb4f91c369a1c6835 (DEFAULT_TABLE table=default.Table_02dcf1af493a4be5857c31c1c50836a7 [id=e9dbf1dbf63741e2b7f8a051bead5973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.539572 752 tablet_service.cc:1467] Processing CreateTablet for tablet ede0928ae7d6441f885e4508bd5f3550 (DEFAULT_TABLE table=default.Table_91a2412f46784a6d870afa172bf3849e [id=513dfc40efff4bff977990c559badf66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.540798 454 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 1218f0bdb1e14bd78d76c6a06805f38c (table default.Table_3e67bbd220754aac89a7b142edcb206c [id=2fb684a787e4420aa9218b369eba5cf2]). Expected version 2 got 0
I20250115 14:52:14.541121 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f9945c40624c3aafda52909edcf467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.550980 453 master_service.cc:946] ReplaceTablet: received request to replace tablet c9a5593068d54d7bb3874cf434505e73 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:14.571091 1018 tablet_service.cc:1467] Processing CreateTablet for tablet ddb66753bb00448c8030b0e0ff03bb22 (DEFAULT_TABLE table=default.Table_f0e1896dc4ac463398d14176e3deda35 [id=3e6b4b42223541dba39ab730ab375fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.571514 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb66753bb00448c8030b0e0ff03bb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.573691 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bf648bb656634def8189fb514a07e144 (table default.Table_fd84fd3d47974e05a580fae352838c20 [id=a00115ecf6954f83b3118654f98e629d]). Expected version 1 got 0
I20250115 14:52:14.574153 446 catalog_manager.cc:5526] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:14.574514 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 5df0250d08214a9783c392071126ed4a (table default.Table_90c834d5ef634e6aa797dd220e3d5faa [id=8ffbe34c2a11417f8ca50dbdd627bf22]). Expected version 2 got 0
I20250115 14:52:14.574672 446 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b (table default.Table_f634603ea42b4215b8ef3df8c446ada7 [id=174b82e1e06f4073a74a25aad8fc8045]). Expected version 2 got 0
W20250115 14:52:14.575467 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a739d36ef9d54c089667ff32de875af6 consensus state has no leader
I20250115 14:52:14.575453 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 45db0bcc1e11481492aea4598e88cfd6 (DEFAULT_TABLE table=default.Table_fff787810e9e4b10aa7c90a8f936a7e6 [id=3962a88382b340398dfac8520f217bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.575685 452 master_service.cc:946] ReplaceTablet: received request to replace tablet 1298fdd0217e4df6bfbafc49b2f2c65c from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:14.575670 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 17405ac097384d62a640e255f1b53649 (DEFAULT_TABLE table=default.Table_8a480bbe40e04cbcbcd51a7e214bfd42 [id=59eed9250dc348feb194acb0301789b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.575788 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45db0bcc1e11481492aea4598e88cfd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.575955 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17405ac097384d62a640e255f1b53649. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:14.576776 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7718729182a340fe9b2dfea5a8321c1f consensus state has no leader
W20250115 14:52:14.577107 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5882252383ba46aba0ec053fab1fd383 consensus state has no leader
I20250115 14:52:14.577417 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3e834623f9c7482d965afd23c89a2510
I20250115 14:52:14.577739 447 catalog_manager.cc:6596] ReplaceTablet: tablet 3e834623f9c7482d965afd23c89a2510 deleted and replaced by tablet 0956531fdeb340208ad3ccab5e934a18
I20250115 14:52:14.580049 2273 consensus_queue.cc:1035] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:14.580808 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet abefc717e0eb41b18725ef0c73154fdd
I20250115 14:52:14.582114 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_58d9a1abb0be4eaf8ebc5c55369a0154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.582669 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58d9a1abb0be4eaf8ebc5c55369a0154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.585107 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_68412e63ed5047c9899420315f0314b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.585546 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68412e63ed5047c9899420315f0314b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.587533 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_bdb6b9f92fac4cb49ce119752a15b45e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:14.588723 1162 tablet_bootstrap.cc:492] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.589861 745 tablet_service.cc:1467] Processing CreateTablet for tablet ed1e5e70acc34f99b4cad1af5db25e80 (DEFAULT_TABLE table=default.Table_dd3cdaff78cf4f24b0ea6ba66ecf8941 [id=cae7df8e8c2b4ded9866bb03e1f6800e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.590658 742 tablet_service.cc:1467] Processing CreateTablet for tablet 417c5441bb7147a98b8ce0cd823a1855 (DEFAULT_TABLE table=default.Table_7c37ffc6335846248f463334008c0fe5 [id=f31b7a822fb84442940c36ac9cd4addc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.590906 749 tablet_service.cc:1467] Processing CreateTablet for tablet c80335eccd3f419989843d7cb1461ab5 (DEFAULT_TABLE table=default.Table_134a6fa6193243e69b7febda51198916 [id=d5c87bbb404f475db2fe69aa76684184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.591099 740 tablet_service.cc:1467] Processing CreateTablet for tablet ddb66753bb00448c8030b0e0ff03bb22 (DEFAULT_TABLE table=default.Table_f0e1896dc4ac463398d14176e3deda35 [id=3e6b4b42223541dba39ab730ab375fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.591468 739 tablet_service.cc:1467] Processing CreateTablet for tablet 45db0bcc1e11481492aea4598e88cfd6 (DEFAULT_TABLE table=default.Table_fff787810e9e4b10aa7c90a8f936a7e6 [id=3962a88382b340398dfac8520f217bb5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:14.591614 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdb6b9f92fac4cb49ce119752a15b45e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.594341 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5392236e7689487c8c5909b627a06a12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.595009 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5392236e7689487c8c5909b627a06a12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.588888 1162 ts_tablet_manager.cc:1397] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.001s
I20250115 14:52:14.597620 747 tablet_service.cc:1467] Processing CreateTablet for tablet 17405ac097384d62a640e255f1b53649 (DEFAULT_TABLE table=default.Table_8a480bbe40e04cbcbcd51a7e214bfd42 [id=59eed9250dc348feb194acb0301789b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.598110 1162 raft_consensus.cc:357] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.598278 1162 raft_consensus.cc:383] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.598320 1162 raft_consensus.cc:738] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.598492 1162 consensus_queue.cc:260] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.598920 1162 ts_tablet_manager.cc:1428] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:14.599175 1162 tablet_bootstrap.cc:492] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.601563 751 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2dd420a0154495bdc76831c20b4797 (DEFAULT_TABLE table=default.Table_b134a43384d144c3a09a6a28068991ae [id=4b92c78d701c4509a6e939737bde052d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.603930 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2dd420a0154495bdc76831c20b4797 (DEFAULT_TABLE table=default.Table_b134a43384d144c3a09a6a28068991ae [id=4b92c78d701c4509a6e939737bde052d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.604336 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b2dd420a0154495bdc76831c20b4797. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.614042 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1298fdd0217e4df6bfbafc49b2f2c65c
I20250115 14:52:14.614146 454 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_e3a17fa2f348459e90b24f586229b7e1 [id=8725c85d30684db495ec7a20962fd75d] alter complete (version 2)
I20250115 14:52:14.614454 452 catalog_manager.cc:6596] ReplaceTablet: tablet 1298fdd0217e4df6bfbafc49b2f2c65c deleted and replaced by tablet 0fb0195969e44d8d9d770c8b0d9c85bf
I20250115 14:52:14.615794 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 313364e8be824a0d9ade9dc0b5164fa7 (DEFAULT_TABLE table=default.Table_65ffb14a8a6d47a48387d591aae1370d [id=351f56e5c35048fba83c82f26a9499f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.616205 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 313364e8be824a0d9ade9dc0b5164fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.616616 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c9a5593068d54d7bb3874cf434505e73
I20250115 14:52:14.616935 453 catalog_manager.cc:6596] ReplaceTablet: tablet c9a5593068d54d7bb3874cf434505e73 deleted and replaced by tablet f5aa92f75f69448d9c43cc808c621f68
I20250115 14:52:14.624740 2403 raft_consensus.cc:491] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.624890 2403 raft_consensus.cc:513] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.625464 2403 leader_election.cc:290] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.626415 770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4adf1353bd554d8294a7f570931095d2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.626783 776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.627713 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1218f0bdb1e14bd78d76c6a06805f38c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.628070 775 raft_consensus.cc:3054] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:14.629187 984 leader_election.cc:343] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:14.629391 984 leader_election.cc:343] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.630219 773 raft_consensus.cc:3054] T dc363f70f3f44d309f69fac9356c6257 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.631438 1162 tablet_bootstrap.cc:654] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.635612 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417c5441bb7147a98b8ce0cd823a1855. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.635861 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b2dd420a0154495bdc76831c20b4797. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.636088 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17405ac097384d62a640e255f1b53649. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.636229 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c80335eccd3f419989843d7cb1461ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.636643 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f9945c40624c3aafda52909edcf467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.636873 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d03eb225fad7459a955a3ec5e0002be1 (table default.Table_a5c4980506e643c7918b795c31ebfbb0 [id=f61735ea21334d049d389d7376ac3fe1]). Expected version 1 got 0
I20250115 14:52:14.639490 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29da671e4f6496b9eb46cac73a1c32b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.640411 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d1f35f4e1f4c2b8d9516df0586ea44. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.641988 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e25f23f07b65483aa5d163d8017a19f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.642505 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e25f23f07b65483aa5d163d8017a19f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.644411 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b940dce7de14c1b844f8835d436a3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.644559 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37970a1f9073400ebf973c4ac3510a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.647185 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f9f9d5c9924479a86b8e2d22e0bf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.647404 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.648468 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e115eb01814dcbb4f91c369a1c6835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.651316 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede0928ae7d6441f885e4508bd5f3550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.654168 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed1e5e70acc34f99b4cad1af5db25e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.656484 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb66753bb00448c8030b0e0ff03bb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.659575 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45db0bcc1e11481492aea4598e88cfd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.660392 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41d8ab30f974bdcba94346460b504a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.661700 1162 tablet_bootstrap.cc:492] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.661828 1162 ts_tablet_manager.cc:1397] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.000s
I20250115 14:52:14.663403 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efaa97b9c88475db3657f4136869d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.666204 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d62f1dfe58874fa39def65ea38c791e9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:14.671216 984 leader_election.cc:343] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.671802 1162 raft_consensus.cc:357] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.672003 1162 raft_consensus.cc:383] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.672060 1162 raft_consensus.cc:738] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.672255 1162 consensus_queue.cc:260] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.672482 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 1298fdd0217e4df6bfbafc49b2f2c65c with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0195969e44d8d9d770c8b0d9c85bf) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.672776 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a5593068d54d7bb3874cf434505e73 with delete_type TABLET_DATA_DELETED (replaced by tablet f5aa92f75f69448d9c43cc808c621f68) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.673213 1025 tablet_service.cc:1467] Processing CreateTablet for tablet cda6870240d24eedaf9ddf57794b880e (DEFAULT_TABLE table=default.Table_dbb09336b4004c35b3ee4880598b02d8 [id=42f0e3311f664194b8347878ac195ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.673705 1014 tablet_service.cc:1467] Processing CreateTablet for tablet bdfb9bf543364b8682d92c58244dc086 (DEFAULT_TABLE table=default.Table_e9629998fb1e463cad667d917dd31dee [id=3d10de88fa894b0c85fe80d2dfabbd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.674057 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet ba33eb3f5a40450db5cf115fbe8a17e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.674449 1012 tablet_service.cc:1467] Processing CreateTablet for tablet c69a14c9c0ab40b999455e6fa7199a91 (DEFAULT_TABLE table=default.Table_c02ac9ab9ef347c0ab6f30ececeb28a1 [id=8cfe1329416b4b03bf910eb0c2916817]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.674950 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cda6870240d24eedaf9ddf57794b880e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.678020 1751 ts_tablet_manager.cc:1918] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.678114 1751 log.cc:1198] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6308f68e41554470b7d1321c60d4eee9
I20250115 14:52:14.680083 1027 tablet_service.cc:1467] Processing CreateTablet for tablet bebb5e3da6c94cee91c430a287fa5e58 (DEFAULT_TABLE table=default.Table_dd78a3cf930047c2b4181c0498fa2f28 [id=94015a18ae034c9f9e918213dd817339]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.680524 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69a14c9c0ab40b999455e6fa7199a91. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:14.680872 984 consensus_peers.cc:487] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:14.682216 1162 ts_tablet_manager.cc:1428] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250115 14:52:14.682794 1162 tablet_bootstrap.cc:492] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.683626 738 tablet_service.cc:1467] Processing CreateTablet for tablet 313364e8be824a0d9ade9dc0b5164fa7 (DEFAULT_TABLE table=default.Table_65ffb14a8a6d47a48387d591aae1370d [id=351f56e5c35048fba83c82f26a9499f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.683945 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 313364e8be824a0d9ade9dc0b5164fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.684437 1162 tablet_bootstrap.cc:654] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.685628 750 tablet_service.cc:1467] Processing CreateTablet for tablet bebb5e3da6c94cee91c430a287fa5e58 (DEFAULT_TABLE table=default.Table_dd78a3cf930047c2b4181c0498fa2f28 [id=94015a18ae034c9f9e918213dd817339]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.686040 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bebb5e3da6c94cee91c430a287fa5e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.686601 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bebb5e3da6c94cee91c430a287fa5e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.690013 753 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a4b3b1b9548018a30d29a552c2eca (DEFAULT_TABLE table=default.Table_5e22d90ab7004e6fb221c1e26c5f564f [id=c32a8f3a2fd74013b3c64c420ea78081]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.690371 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6a4b3b1b9548018a30d29a552c2eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.691284 1751 ts_tablet_manager.cc:1939] T 6308f68e41554470b7d1321c60d4eee9 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:14.692576 1010 tablet_service.cc:1467] Processing CreateTablet for tablet c80335eccd3f419989843d7cb1461ab5 (DEFAULT_TABLE table=default.Table_134a6fa6193243e69b7febda51198916 [id=d5c87bbb404f475db2fe69aa76684184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.692900 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c80335eccd3f419989843d7cb1461ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.695484 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet abefc717e0eb41b18725ef0c73154fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.695900 1020 tablet_service.cc:1467] Processing CreateTablet for tablet ed1e5e70acc34f99b4cad1af5db25e80 (DEFAULT_TABLE table=default.Table_dd3cdaff78cf4f24b0ea6ba66ecf8941 [id=cae7df8e8c2b4ded9866bb03e1f6800e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.697289 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed1e5e70acc34f99b4cad1af5db25e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.698858 752 tablet_service.cc:1514] Processing DeleteTablet for tablet abefc717e0eb41b18725ef0c73154fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.696604 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 417c5441bb7147a98b8ce0cd823a1855 (DEFAULT_TABLE table=default.Table_7c37ffc6335846248f463334008c0fe5 [id=f31b7a822fb84442940c36ac9cd4addc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.700574 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417c5441bb7147a98b8ce0cd823a1855. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.700708 752 tablet_service.cc:1514] Processing DeleteTablet for tablet 1298fdd0217e4df6bfbafc49b2f2c65c with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0195969e44d8d9d770c8b0d9c85bf) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.703580 1162 tablet_bootstrap.cc:492] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.703708 1162 ts_tablet_manager.cc:1397] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20250115 14:52:14.704409 1162 raft_consensus.cc:357] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.704579 1162 raft_consensus.cc:383] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.704629 1162 raft_consensus.cc:738] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.704809 1162 consensus_queue.cc:260] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.705257 1162 ts_tablet_manager.cc:1428] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250115 14:52:14.705996 1162 tablet_bootstrap.cc:492] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.708564 1162 tablet_bootstrap.cc:654] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.691797 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e834623f9c7482d965afd23c89a2510 with delete_type TABLET_DATA_DELETED (replaced by tablet 0956531fdeb340208ad3ccab5e934a18) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.696278 1021 tablet_service.cc:1467] Processing CreateTablet for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e (DEFAULT_TABLE table=default.Table_06ae3b8ba32f4b05af77d455a746ecd2 [id=6ac47a59a5f249d48f4d69f7bb28f02c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.696943 1015 tablet_service.cc:1467] Processing CreateTablet for tablet ede0928ae7d6441f885e4508bd5f3550 (DEFAULT_TABLE table=default.Table_91a2412f46784a6d870afa172bf3849e [id=513dfc40efff4bff977990c559badf66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.713932 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1a9d1352e9f0419582e92a1faaf29288" } modify_external_catalogs: true
I20250115 14:52:14.714120 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1a9d1352e9f0419582e92a1faaf29288" } modify_external_catalogs: true
I20250115 14:52:14.714612 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.715000 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91" } modify_external_catalogs: true
I20250115 14:52:14.715160 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0a6d7c2a9fa54a0f968167c8bfd89f91" } modify_external_catalogs: true
I20250115 14:52:14.716449 2273 raft_consensus.cc:491] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.716562 2273 raft_consensus.cc:513] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.717094 2273 leader_election.cc:290] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.717633 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba33eb3f5a40450db5cf115fbe8a17e7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:14.718215 984 leader_election.cc:343] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.720927 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a4b3b1b9548018a30d29a552c2eca (DEFAULT_TABLE table=default.Table_5e22d90ab7004e6fb221c1e26c5f564f [id=c32a8f3a2fd74013b3c64c420ea78081]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.725364 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e834623f9c7482d965afd23c89a2510 with delete_type TABLET_DATA_DELETED (replaced by tablet 0956531fdeb340208ad3ccab5e934a18) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.727079 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5b33c876fab1476aba3b1f6e799fa96c
I20250115 14:52:14.728470 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet db57f7c3a4c14cc19d3a0624e5b1817a
I20250115 14:52:14.731554 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet db57f7c3a4c14cc19d3a0624e5b1817a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.734038 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_aa12615b9ec74e788e7883960ac22d2a" } modify_external_catalogs: true
I20250115 14:52:14.734205 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_aa12615b9ec74e788e7883960ac22d2a" } modify_external_catalogs: true
I20250115 14:52:14.734272 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6a4b3b1b9548018a30d29a552c2eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.738083 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_095871dcaa2547f8a7d859237e9bf4fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:14.738181 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_67c82d5293bd4157bc5887a9d6410293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.738605 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_095871dcaa2547f8a7d859237e9bf4fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.738828 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c0c0ea95429d42b58100f5fd5f95aabb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.739168 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0c0ea95429d42b58100f5fd5f95aabb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.739900 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfb9bf543364b8682d92c58244dc086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.740195 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet e6978223c1984a38bec5f2d06a9cf457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:14.738613 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67c82d5293bd4157bc5887a9d6410293 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.743544 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9591925055134e48ad3592198934e708" } modify_external_catalogs: true
I20250115 14:52:14.743686 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9591925055134e48ad3592198934e708" } modify_external_catalogs: true
I20250115 14:52:14.744355 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ee7134670e574f279ad4f5557f395e69" } new_table_name: "default.Table_1086760a509c459d81e3cc1a62e78111" modify_external_catalogs: true
I20250115 14:52:14.745934 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b33c876fab1476aba3b1f6e799fa96c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.753273 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede0928ae7d6441f885e4508bd5f3550. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:14.756455 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69a5e5c5758445d3b97615e82f91fcf7 consensus state has no leader
I20250115 14:52:14.764448 1162 tablet_bootstrap.cc:492] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.764673 1162 ts_tablet_manager.cc:1397] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20250115 14:52:14.765580 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 2560d42c15c64b4c9c0cfe95706b4d9e with delete_type TABLET_DATA_DELETED (replaced by tablet f4536bb3d44b40d2825e1f4e59d8a148) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.765869 1162 raft_consensus.cc:357] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.766192 1162 raft_consensus.cc:383] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.766280 1162 raft_consensus.cc:738] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.766517 1162 consensus_queue.cc:260] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.767393 1162 ts_tablet_manager.cc:1428] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:14.769136 1162 tablet_bootstrap.cc:492] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.770377 1164 tablet_bootstrap.cc:492] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:14.770558 1164 ts_tablet_manager.cc:1397] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.290s user 0.031s sys 0.034s
I20250115 14:52:14.771363 1162 tablet_bootstrap.cc:654] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.772153 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1186) took 1473 ms. Trace:
I20250115 14:52:14.772233 1751 rpcz_store.cc:276] 0115 14:52:13.298354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.298398 (+ 44us) service_pool.cc:225] Handling call
0115 14:52:14.398505 (+1100107us) ts_tablet_manager.cc:1145] Deleting tablet 6308f68e41554470b7d1321c60d4eee9
0115 14:52:14.398514 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.675312 (+276798us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.677952 (+ 2640us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.772137 (+ 94185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1099847}
I20250115 14:52:14.772568 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1187) took 1261 ms. Trace:
I20250115 14:52:14.772629 1751 rpcz_store.cc:276] 0115 14:52:13.511602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.511655 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:14.772466 (+1260811us) ts_tablet_manager.cc:1145] Deleting tablet c497fb29752d4ed7bda207301f90cf05
0115 14:52:14.772474 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.772556 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1253784}
I20250115 14:52:14.772876 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1188) took 1260 ms. Trace:
I20250115 14:52:14.772928 1751 rpcz_store.cc:276] 0115 14:52:13.512102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.512152 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:14.772798 (+1260646us) ts_tablet_manager.cc:1145] Deleting tablet 2c89d5dd5dd74df2a66e50636cc5e7f9
0115 14:52:14.772806 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.772865 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254057}
I20250115 14:52:14.773126 1751 tablet_replica.cc:331] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:14.773283 1751 raft_consensus.cc:2238] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.773401 1751 raft_consensus.cc:2267] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.773845 1751 ts_tablet_manager.cc:1905] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.779626 1751 ts_tablet_manager.cc:1918] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.779734 1751 log.cc:1198] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6e2d55a4e32448fbb98f704dbf1657b9
I20250115 14:52:14.779994 1751 ts_tablet_manager.cc:1939] T 6e2d55a4e32448fbb98f704dbf1657b9 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:14.782615 2273 raft_consensus.cc:491] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.782635 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6308f68e41554470b7d1321c60d4eee9 (table default.Table_33f4705d7a1b474d82734fe4a80c0424 [id=b0ebc66f46364e51b73bdee697f00b62]) successfully deleted
I20250115 14:52:14.782739 2273 raft_consensus.cc:513] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:14.782841 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c497fb29752d4ed7bda207301f90cf05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c497fb29752d4ed7bda207301f90cf05 already in progress: creating tablet
W20250115 14:52:14.782982 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c89d5dd5dd74df2a66e50636cc5e7f9 already in progress: creating tablet
I20250115 14:52:14.783049 1164 raft_consensus.cc:357] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.783233 1164 raft_consensus.cc:383] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.783285 1164 raft_consensus.cc:738] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.783321 2273 leader_election.cc:290] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.783466 1164 consensus_queue.cc:260] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.784435 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36de948083a946eba1a96f276e4f4f91" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:14.786746 984 leader_election.cc:343] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.787914 1162 tablet_bootstrap.cc:492] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:14.788038 1162 ts_tablet_manager.cc:1397] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.001s
I20250115 14:52:14.788180 1164 ts_tablet_manager.cc:1428] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250115 14:52:14.788903 1164 tablet_bootstrap.cc:492] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:14.788993 1162 raft_consensus.cc:357] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.789161 1162 raft_consensus.cc:383] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.789201 1162 raft_consensus.cc:738] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.789606 1162 consensus_queue.cc:260] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.790405 1162 ts_tablet_manager.cc:1428] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:14.791025 1162 tablet_bootstrap.cc:492] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:14.793152 1162 tablet_bootstrap.cc:654] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.803176 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d050f7c481a549248abc7c8beb08de6e" } new_table_name: "default.Table_6ab2a32317d84ef0bc98f61078e58387" modify_external_catalogs: true
I20250115 14:52:14.805367 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1bef4c655e464405a4b2893dba5c9eca" } new_table_name: "default.Table_df1ed1e4b434476aa9a1c7852afb0cd2" modify_external_catalogs: true
I20250115 14:52:14.805415 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_548ea14243474f25b73cfc9d56a806b3" } new_table_name: "default.Table_6881a815c7784e768e4a83edfe0b5a3f" modify_external_catalogs: true
I20250115 14:52:14.806473 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b71be8b255d34a7b9b1e3fd33dc4fb8c" } new_table_name: "default.Table_54c2a74496ee4d64847d93cc24a23e6e" modify_external_catalogs: true
I20250115 14:52:14.807412 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1fe291920627495bb359337954580862" } modify_external_catalogs: true
I20250115 14:52:14.807538 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1fe291920627495bb359337954580862" } modify_external_catalogs: true
W20250115 14:52:14.809391 710 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:36351, user_credentials={real_user=slave}} blocked reactor thread for 184432us
I20250115 14:52:14.816905 1164 tablet_bootstrap.cc:654] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.462769 882 tablet_service.cc:1467] Processing CreateTablet for tablet afea664025e54f6099d825c8f7a0ea1a (DEFAULT_TABLE table=default.Table_1033f77f38414857a1255c05f93935d2 [id=b079ea0cfd244f0abda1500173404ca3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.817759 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afea664025e54f6099d825c8f7a0ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.817831 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f7d31cfc0fe54ffdb1b9fc0b8b26fe3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:14.817909 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9480f0c225c34cef9d44fc404aa3cefd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.818334 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9480f0c225c34cef9d44fc404aa3cefd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:14.818341 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7d31cfc0fe54ffdb1b9fc0b8b26fe3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.483408 909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4adf1353bd554d8294a7f570931095d2" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.489444 884 tablet_service.cc:1467] Processing CreateTablet for tablet 95f9f9d5c9924479a86b8e2d22e0bf14 (DEFAULT_TABLE table=default.Table_56d30c076381477494fb1b08ef4d4c46 [id=da1ebfcda8a043a9891fd3e0aaa24e19]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:14.820609 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7baacab95e6c4166b3437a9448306088 consensus state has no leader
W20250115 14:52:14.820729 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23ad6f98f8fa445ab2100f29a7ffb991 consensus state has no leader
I20250115 14:52:14.820801 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f9f9d5c9924479a86b8e2d22e0bf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.489776 881 tablet_service.cc:1467] Processing CreateTablet for tablet 7b940dce7de14c1b844f8835d436a3d8 (DEFAULT_TABLE table=default.Table_7584e090790949b3b7d23a1ced5078be [id=3872d9b0d70547f79ca6cb24ea506946]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.821126 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b940dce7de14c1b844f8835d436a3d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.822207 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
I20250115 14:52:14.493904 883 tablet_service.cc:1467] Processing CreateTablet for tablet 58d1f35f4e1f4c2b8d9516df0586ea44 (DEFAULT_TABLE table=default.Table_554ebf7919a940628afd939485c575d9 [id=e75dfaeb216844d082092fe0d36e10cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.824556 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58d1f35f4e1f4c2b8d9516df0586ea44. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:14.824518 983 leader_election.cc:343] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.824681 983 leader_election.cc:304] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.825170 2563 raft_consensus.cc:2743] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.827131 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7e70288eb248f1a71646c66e376a63" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:14.829493 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3daf6416ad847aeaa83087cc81c9239 consensus state has no leader
I20250115 14:52:14.494197 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 061234cb76b842a7a374ddd5f44fd9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.830149 893 tablet_service.cc:1467] Processing CreateTablet for tablet c80335eccd3f419989843d7cb1461ab5 (DEFAULT_TABLE table=default.Table_134a6fa6193243e69b7febda51198916 [id=d5c87bbb404f475db2fe69aa76684184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.494227 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.495961 891 tablet_service.cc:1467] Processing CreateTablet for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e (DEFAULT_TABLE table=default.Table_06ae3b8ba32f4b05af77d455a746ecd2 [id=6ac47a59a5f249d48f4d69f7bb28f02c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.501765 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad3cda1056349a789a1751c5d386d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.501876 887 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a4837b72eb4008ad612a9fc86dfac7 with delete_type TABLET_DATA_DELETED (replaced by tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.830803 887 tablet_service.cc:1467] Processing CreateTablet for tablet 45db0bcc1e11481492aea4598e88cfd6 (DEFAULT_TABLE table=default.Table_fff787810e9e4b10aa7c90a8f936a7e6 [id=3962a88382b340398dfac8520f217bb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.502022 913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1218f0bdb1e14bd78d76c6a06805f38c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.502182 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 24729156769443f8926577bae7707847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.521075 878 tablet_service.cc:1467] Processing CreateTablet for tablet d62f1dfe58874fa39def65ea38c791e9 (DEFAULT_TABLE table=default.Table_7c016ac18c7c4026b8e60a53f889bf0d [id=dbaa86e0c6294e8aba837f930d95b144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.830803 889 tablet_service.cc:1467] Processing CreateTablet for tablet ddb66753bb00448c8030b0e0ff03bb22 (DEFAULT_TABLE table=default.Table_f0e1896dc4ac463398d14176e3deda35 [id=3e6b4b42223541dba39ab730ab375fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.831300 876 tablet_service.cc:1467] Processing CreateTablet for tablet 17405ac097384d62a640e255f1b53649 (DEFAULT_TABLE table=default.Table_8a480bbe40e04cbcbcd51a7e214bfd42 [id=59eed9250dc348feb194acb0301789b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.521100 890 tablet_service.cc:1467] Processing CreateTablet for tablet 05f9945c40624c3aafda52909edcf467 (DEFAULT_TABLE table=default.Table_c586d1fdfee24db2b1b3e9196fca1641 [id=1d95d1ca67fc4239abd896a78a80fb39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.521765 892 tablet_service.cc:1467] Processing CreateTablet for tablet e41d8ab30f974bdcba94346460b504a6 (DEFAULT_TABLE table=default.Table_c3d7415d4479485b9629973554670e05 [id=36e4935a4a1f4dc995e9c7bf314e0ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.552910 879 tablet_service.cc:1467] Processing CreateTablet for tablet a29da671e4f6496b9eb46cac73a1c32b (DEFAULT_TABLE table=default.Table_b322d270a8f74d2e95bb772545cd9ade [id=0a734bb5c8fa47d099cb23c1f20f2716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.553009 874 tablet_service.cc:1467] Processing CreateTablet for tablet 37970a1f9073400ebf973c4ac3510a72 (DEFAULT_TABLE table=default.Table_6c72bc6feb014b768d58cf8c6095fa8a [id=ce6ac893855e4cfb9a57ffd197bb938a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.553300 880 tablet_service.cc:1467] Processing CreateTablet for tablet 3efaa97b9c88475db3657f4136869d9c (DEFAULT_TABLE table=default.Table_1d7ea0f078e148088d47b6e1b3b1903b [id=c10b4b90d6134ae392d22d32f832f839]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.553536 875 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc (DEFAULT_TABLE table=default.Table_c49292ffc41542abb7c1104187ce5cba [id=085b32beb8424ec98e18d011609b6629]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.553817 885 tablet_service.cc:1467] Processing CreateTablet for tablet a0e115eb01814dcbb4f91c369a1c6835 (DEFAULT_TABLE table=default.Table_02dcf1af493a4be5857c31c1c50836a7 [id=e9dbf1dbf63741e2b7f8a051bead5973]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.556008 2669 raft_consensus.cc:491] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.831871 2669 raft_consensus.cc:513] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.832468 2669 leader_election.cc:290] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.563776 877 tablet_service.cc:1467] Processing CreateTablet for tablet ede0928ae7d6441f885e4508bd5f3550 (DEFAULT_TABLE table=default.Table_91a2412f46784a6d870afa172bf3849e [id=513dfc40efff4bff977990c559badf66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.564724 888 tablet_service.cc:1467] Processing CreateTablet for tablet ed1e5e70acc34f99b4cad1af5db25e80 (DEFAULT_TABLE table=default.Table_dd3cdaff78cf4f24b0ea6ba66ecf8941 [id=cae7df8e8c2b4ded9866bb03e1f6800e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.598417 2549 consensus_queue.cc:1035] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:14.628641 2657 raft_consensus.cc:491] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:52:14.833364 983 leader_election.cc:343] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.718820 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba33eb3f5a40450db5cf115fbe8a17e7" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.833478 983 leader_election.cc:304] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.833488 2657 raft_consensus.cc:513] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.833848 2563 raft_consensus.cc:2743] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:14.721017 1168 tablet_bootstrap.cc:492] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:14.834127 1168 ts_tablet_manager.cc:1397] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.394s user 0.004s sys 0.000s
I20250115 14:52:14.784262 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36de948083a946eba1a96f276e4f4f91" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:14.494088 886 tablet_service.cc:1514] Processing DeleteTablet for tablet 7830c683a109448ab06c15b5de692e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.834510 886 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2dd420a0154495bdc76831c20b4797 (DEFAULT_TABLE table=default.Table_b134a43384d144c3a09a6a28068991ae [id=4b92c78d701c4509a6e939737bde052d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.834831 1168 raft_consensus.cc:357] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.834985 1168 raft_consensus.cc:383] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.835031 1168 raft_consensus.cc:738] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.835181 1168 consensus_queue.cc:260] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.835638 1168 ts_tablet_manager.cc:1428] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250115 14:52:14.836259 911 raft_consensus.cc:2463] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
W20250115 14:52:14.837078 983 leader_election.cc:343] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.837184 983 leader_election.cc:304] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.837635 2563 raft_consensus.cc:2743] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:14.837137 848 consensus_peers.cc:487] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:14.838654 983 leader_election.cc:343] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:14.839027 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7830c683a109448ab06c15b5de692e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7830c683a109448ab06c15b5de692e8b already in progress: creating tablet
I20250115 14:52:14.839006 983 leader_election.cc:304] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:14.839254 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 061234cb76b842a7a374ddd5f44fd9a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 061234cb76b842a7a374ddd5f44fd9a1 already in progress: creating tablet
W20250115 14:52:14.839418 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 0651be54e0b244a4bace9f47efc9b58a because the tablet was not found. No further retry: Not found: Tablet not found: 0651be54e0b244a4bace9f47efc9b58a
W20250115 14:52:14.839596 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9ad3cda1056349a789a1751c5d386d0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ad3cda1056349a789a1751c5d386d0c already in progress: creating tablet
W20250115 14:52:14.839761 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 13a4837b72eb4008ad612a9fc86dfac7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13a4837b72eb4008ad612a9fc86dfac7 already in progress: creating tablet
I20250115 14:52:14.839815 2563 raft_consensus.cc:2743] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:14.839921 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 24729156769443f8926577bae7707847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24729156769443f8926577bae7707847 already in progress: creating tablet
I20250115 14:52:14.840839 983 leader_election.cc:304] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.841375 2563 raft_consensus.cc:2798] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:14.841444 2563 raft_consensus.cc:491] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:14.841501 2563 raft_consensus.cc:3054] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.842089 451 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 8c7e70288eb248f1a71646c66e376a63 (table default.Table_2ea7cdeffde54825be02af36704833a0 [id=13d5cd4ccf2245319f40495141007b58]). Expected version 1 got 0
I20250115 14:52:14.843925 2563 raft_consensus.cc:513] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:14.844451 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7da4a558e1ea4bccb868ad5c489e96d5 consensus state has no leader
I20250115 14:52:14.845835 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:14.846393 984 leader_election.cc:343] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:14.846453 2563 leader_election.cc:290] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:14.846689 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c80335eccd3f419989843d7cb1461ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.846858 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed1e5e70acc34f99b4cad1af5db25e80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.847311 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a29da671e4f6496b9eb46cac73a1c32b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.847635 881 tablet_service.cc:1467] Processing CreateTablet for tablet 313364e8be824a0d9ade9dc0b5164fa7 (DEFAULT_TABLE table=default.Table_65ffb14a8a6d47a48387d591aae1370d [id=351f56e5c35048fba83c82f26a9499f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.847939 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 313364e8be824a0d9ade9dc0b5164fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.848373 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45db0bcc1e11481492aea4598e88cfd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.848423 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:14.848570 911 raft_consensus.cc:3054] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.850968 911 raft_consensus.cc:2463] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:14.851208 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37970a1f9073400ebf973c4ac3510a72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.851356 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d62f1dfe58874fa39def65ea38c791e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.851492 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f9945c40624c3aafda52909edcf467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.852418 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ede0928ae7d6441f885e4508bd5f3550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.859369 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17405ac097384d62a640e255f1b53649. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.859575 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb66753bb00448c8030b0e0ff03bb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.859828 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3efaa97b9c88475db3657f4136869d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.861255 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.864462 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e41d8ab30f974bdcba94346460b504a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.868417 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e115eb01814dcbb4f91c369a1c6835. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.869681 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e834623f9c7482d965afd23c89a2510 with delete_type TABLET_DATA_DELETED (replaced by tablet 0956531fdeb340208ad3ccab5e934a18) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.869853 892 tablet_service.cc:1514] Processing DeleteTablet for tablet abefc717e0eb41b18725ef0c73154fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.869956 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 1298fdd0217e4df6bfbafc49b2f2c65c with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0195969e44d8d9d770c8b0d9c85bf) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.870044 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 1140016d11a74a2cadba494c0b319c54 with delete_type TABLET_DATA_DELETED (replaced by tablet 068771fee92f403d8997646291a83dc6) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.870210 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 325cd3fd8b4d447998822163e575e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.870251 983 leader_election.cc:304] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:14.870329 891 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a5593068d54d7bb3874cf434505e73 with delete_type TABLET_DATA_DELETED (replaced by tablet f5aa92f75f69448d9c43cc808c621f68) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.870594 891 tablet_service.cc:1467] Processing CreateTablet for tablet bdfb9bf543364b8682d92c58244dc086 (DEFAULT_TABLE table=default.Table_e9629998fb1e463cad667d917dd31dee [id=3d10de88fa894b0c85fe80d2dfabbd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.870667 2563 raft_consensus.cc:2798] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:14.871078 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfb9bf543364b8682d92c58244dc086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.871173 1168 tablet_bootstrap.cc:492] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:14.871358 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b2dd420a0154495bdc76831c20b4797. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.871522 882 tablet_service.cc:1467] Processing CreateTablet for tablet c69a14c9c0ab40b999455e6fa7199a91 (DEFAULT_TABLE table=default.Table_c02ac9ab9ef347c0ab6f30ececeb28a1 [id=8cfe1329416b4b03bf910eb0c2916817]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.871794 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69a14c9c0ab40b999455e6fa7199a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.872494 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f4a0c8392cf44eeaa27cdcaf43f9edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.874778 1168 tablet_bootstrap.cc:654] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:14.828723 884 tablet_service.cc:1467] Processing CreateTablet for tablet 417c5441bb7147a98b8ce0cd823a1855 (DEFAULT_TABLE table=default.Table_7c37ffc6335846248f463334008c0fe5 [id=f31b7a822fb84442940c36ac9cd4addc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.880456 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417c5441bb7147a98b8ce0cd823a1855. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.880650 876 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b33c876fab1476aba3b1f6e799fa96c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.880884 876 tablet_service.cc:1514] Processing DeleteTablet for tablet db57f7c3a4c14cc19d3a0624e5b1817a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.859869 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c0e6fbc56b8545359a96da934e8410ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:14.858685 883 tablet_service.cc:1467] Processing CreateTablet for tablet cda6870240d24eedaf9ddf57794b880e (DEFAULT_TABLE table=default.Table_dbb09336b4004c35b3ee4880598b02d8 [id=42f0e3311f664194b8347878ac195ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.885926 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cda6870240d24eedaf9ddf57794b880e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.833967 2657 leader_election.cc:290] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.858238 878 tablet_service.cc:1467] Processing CreateTablet for tablet 3e6a4b3b1b9548018a30d29a552c2eca (DEFAULT_TABLE table=default.Table_5e22d90ab7004e6fb221c1e26c5f564f [id=c32a8f3a2fd74013b3c64c420ea78081]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.886493 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6a4b3b1b9548018a30d29a552c2eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.886703 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cce291ff531b4d4dbf028cbdb72cc021"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:14.887622 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_adcc45a64d8840fba4e22623a802abb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:14.852278 888 tablet_service.cc:1467] Processing CreateTablet for tablet bebb5e3da6c94cee91c430a287fa5e58 (DEFAULT_TABLE table=default.Table_dd78a3cf930047c2b4181c0498fa2f28 [id=94015a18ae034c9f9e918213dd817339]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.888139 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bebb5e3da6c94cee91c430a287fa5e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.888247 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_883692e629d34120b90cdbf0744cdfc8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.888656 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_883692e629d34120b90cdbf0744cdfc8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.889034 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
W20250115 14:52:14.889313 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adcc45a64d8840fba4e22623a802abb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.829769 2747 raft_consensus.cc:491] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.889876 2747 raft_consensus.cc:513] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.890947 2747 leader_election.cc:290] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:14.891777 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cce291ff531b4d4dbf028cbdb72cc021 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.870059 892 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4475441e4a467a8d593b742d2a5b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:14.894758 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_961a6a3237c54002bab6a3b30ca9301e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.898140 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0e6fbc56b8545359a96da934e8410ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.900107 1167 log.cc:1198] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/248cadcdef8b4e99923cdbca39690685
W20250115 14:52:14.900146 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet abefc717e0eb41b18725ef0c73154fdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abefc717e0eb41b18725ef0c73154fdd already in progress: creating tablet
W20250115 14:52:14.900396 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1298fdd0217e4df6bfbafc49b2f2c65c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1298fdd0217e4df6bfbafc49b2f2c65c already in progress: creating tablet
I20250115 14:52:14.900492 1167 ts_tablet_manager.cc:1939] T 248cadcdef8b4e99923cdbca39690685 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:14.900766 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1201) took 1037 ms. Trace:
I20250115 14:52:14.900856 1167 rpcz_store.cc:276] 0115 14:52:13.862859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.875634 (+ 12775us) service_pool.cc:225] Handling call
0115 14:52:14.449373 (+573739us) ts_tablet_manager.cc:1145] Deleting tablet 248cadcdef8b4e99923cdbca39690685
0115 14:52:14.449382 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.456321 (+ 6939us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.459157 (+ 2836us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.900753 (+441596us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":573420}
W20250115 14:52:14.901014 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 3e834623f9c7482d965afd23c89a2510 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e834623f9c7482d965afd23c89a2510 already in progress: creating tablet
I20250115 14:52:14.901862 1167 tablet_replica.cc:331] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
W20250115 14:52:14.902035 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_961a6a3237c54002bab6a3b30ca9301e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.902097 1167 raft_consensus.cc:2238] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.902745 1014 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:14.907737 750 tablet_service.cc:1467] Processing CreateTablet for tablet cda6870240d24eedaf9ddf57794b880e (DEFAULT_TABLE table=default.Table_dbb09336b4004c35b3ee4880598b02d8 [id=42f0e3311f664194b8347878ac195ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.908191 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cda6870240d24eedaf9ddf57794b880e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.908499 753 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a5593068d54d7bb3874cf434505e73 with delete_type TABLET_DATA_DELETED (replaced by tablet f5aa92f75f69448d9c43cc808c621f68) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.911008 747 tablet_service.cc:1467] Processing CreateTablet for tablet c69a14c9c0ab40b999455e6fa7199a91 (DEFAULT_TABLE table=default.Table_c02ac9ab9ef347c0ab6f30ececeb28a1 [id=8cfe1329416b4b03bf910eb0c2916817]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.911358 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69a14c9c0ab40b999455e6fa7199a91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.911455 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 248cadcdef8b4e99923cdbca39690685 (table default.Table_b64c63a30a1c4f328986efc44022083b [id=5d393887a5e54536a4139752211d21a1]) successfully deleted
W20250115 14:52:14.911808 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 11aeb6c73b4d42058736ffce551e311c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11aeb6c73b4d42058736ffce551e311c already in progress: creating tablet
W20250115 14:52:14.912104 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet eb78eb2c5b9b44fba1a4e140a8e102d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb78eb2c5b9b44fba1a4e140a8e102d7 already in progress: creating tablet
I20250115 14:52:14.912038 1047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7e70288eb248f1a71646c66e376a63" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:14.912369 1167 raft_consensus.cc:2267] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.912379 1044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8da9bd5ec841ebbc61b0f8de5ace3b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:14.912546 1044 raft_consensus.cc:2463] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:14.912897 1167 ts_tablet_manager.cc:1905] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:14.913172 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 4e459144038d411d9e76322dee8f004b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e459144038d411d9e76322dee8f004b already in progress: creating tablet
I20250115 14:52:14.913434 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.913687 847 leader_election.cc:304] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:52:14.913867 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d36fa3146f0f40a59e5f96cce203fee0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d36fa3146f0f40a59e5f96cce203fee0 already in progress: creating tablet
I20250115 14:52:14.914134 2657 raft_consensus.cc:2798] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:14.914206 2657 raft_consensus.cc:491] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:14.914168 2748 consensus_queue.cc:1035] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:14.914654 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b33c876fab1476aba3b1f6e799fa96c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.914851 741 tablet_service.cc:1514] Processing DeleteTablet for tablet db57f7c3a4c14cc19d3a0624e5b1817a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.915004 740 tablet_service.cc:1514] Processing DeleteTablet for tablet a48227ec5aa846c3ac63bd29ed38be25 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:14.915190 1166 tablet_replica.cc:331] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:14.915364 1166 raft_consensus.cc:2238] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.915493 1166 raft_consensus.cc:2267] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.916152 1166 ts_tablet_manager.cc:1905] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.916121 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1140016d11a74a2cadba494c0b319c54" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:14.914260 2657 raft_consensus.cc:3054] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.916963 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:52:14.917945 847 leader_election.cc:343] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 1140016d11a74a2cadba494c0b319c54
I20250115 14:52:14.918375 738 tablet_service.cc:1467] Processing CreateTablet for tablet bdfb9bf543364b8682d92c58244dc086 (DEFAULT_TABLE table=default.Table_e9629998fb1e463cad667d917dd31dee [id=3d10de88fa894b0c85fe80d2dfabbd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:14.918761 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfb9bf543364b8682d92c58244dc086. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:14.921788 2657 raft_consensus.cc:513] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.922047 1166 ts_tablet_manager.cc:1918] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:14.922144 1166 log.cc:1198] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/a48227ec5aa846c3ac63bd29ed38be25
I20250115 14:52:14.922394 1166 ts_tablet_manager.cc:1939] T a48227ec5aa846c3ac63bd29ed38be25 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:14.923079 2657 leader_election.cc:290] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.926478 2657 raft_consensus.cc:491] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:14.926640 2657 raft_consensus.cc:513] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.927603 2657 leader_election.cc:290] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.932119 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_482c3b93feb244538f5483b3419a7a60" } modify_external_catalogs: true
I20250115 14:52:14.932317 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_482c3b93feb244538f5483b3419a7a60" } modify_external_catalogs: true
I20250115 14:52:14.933857 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b0a8c96252014e9298515a1f5ff14410" } modify_external_catalogs: true
I20250115 14:52:14.934012 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b0a8c96252014e9298515a1f5ff14410" } modify_external_catalogs: true
I20250115 14:52:14.935139 1049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8da9bd5ec841ebbc61b0f8de5ace3b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:14.935336 1049 raft_consensus.cc:3054] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.937834 1049 raft_consensus.cc:2463] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:14.938316 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70425c25919b4d0b999450210e1cb33f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:14.938495 1043 raft_consensus.cc:2463] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:14.939153 847 leader_election.cc:304] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:14.939453 847 leader_election.cc:304] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:14.940716 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1189) took 1428 ms. Trace:
I20250115 14:52:14.940822 1751 rpcz_store.cc:276] 0115 14:52:13.512480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.512527 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:14.773092 (+1260565us) ts_tablet_manager.cc:1145] Deleting tablet 6e2d55a4e32448fbb98f704dbf1657b9
0115 14:52:14.773099 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.776865 (+ 3766us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.779544 (+ 2679us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:14.940699 (+161155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254293}
I20250115 14:52:14.941211 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1190) took 1428 ms. Trace:
I20250115 14:52:14.941301 1751 rpcz_store.cc:276] 0115 14:52:13.512910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.512962 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:14.941097 (+1428135us) ts_tablet_manager.cc:1145] Deleting tablet e6821f4c34f042c8b4819176fe4418ff
0115 14:52:14.941106 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.941196 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1422253}
I20250115 14:52:14.941635 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1191) took 1428 ms. Trace:
I20250115 14:52:14.941717 1751 rpcz_store.cc:276] 0115 14:52:13.513561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.513610 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:14.941531 (+1427921us) ts_tablet_manager.cc:1145] Deleting tablet f1e3879802464acaa279dfe4fbb634bb
0115 14:52:14.941540 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.941621 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1422651}
I20250115 14:52:14.942015 1751 tablet_replica.cc:331] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:14.942232 1751 raft_consensus.cc:2238] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:14.942502 1751 raft_consensus.cc:2267] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:14.943054 1751 ts_tablet_manager.cc:1905] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:14.943413 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6e2d55a4e32448fbb98f704dbf1657b9 (table default.Table_f3c56dc604f2471083ae4303c707c7e3 [id=03cd191267784fac938c73c9b363855a]) successfully deleted
W20250115 14:52:14.943612 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e6821f4c34f042c8b4819176fe4418ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6821f4c34f042c8b4819176fe4418ff already in progress: creating tablet
W20250115 14:52:14.943768 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f1e3879802464acaa279dfe4fbb634bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e3879802464acaa279dfe4fbb634bb already in progress: creating tablet
I20250115 14:52:14.946122 2657 raft_consensus.cc:2798] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:14.946297 2748 raft_consensus.cc:2798] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:14.946348 2748 raft_consensus.cc:491] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:14.946398 2748 raft_consensus.cc:3054] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:14.948393 2657 raft_consensus.cc:695] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:14.948685 2657 consensus_queue.cc:237] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:14.950488 2748 raft_consensus.cc:513] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.951267 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_252a71de31274362a56b0a5b528f14e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:14.951550 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70425c25919b4d0b999450210e1cb33f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:14.951719 1043 raft_consensus.cc:3054] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
W20250115 14:52:14.951849 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_252a71de31274362a56b0a5b528f14e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.952819 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_be4f13e702dc4d6cb81ab23dcfe5203e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.953251 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be4f13e702dc4d6cb81ab23dcfe5203e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:14.954226 1043 raft_consensus.cc:2463] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:14.954717 847 leader_election.cc:304] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:14.954973 2549 raft_consensus.cc:2798] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:14.955070 2549 raft_consensus.cc:695] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:14.955284 2549 consensus_queue.cc:237] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.958720 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_454909c032fc4a9b82dd07b728e001ea" } new_table_name: "default.Table_52bce80521874a99b9965322de01bc4b" modify_external_catalogs: true
I20250115 14:52:14.959419 2748 leader_election.cc:290] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:14.976616 1043 raft_consensus.cc:1270] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:14.977542 2563 raft_consensus.cc:695] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:14.977528 2748 consensus_queue.cc:1035] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:14.977936 2563 consensus_queue.cc:237] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.983685 1168 tablet_bootstrap.cc:492] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:14.983896 1168 ts_tablet_manager.cc:1397] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.113s user 0.007s sys 0.002s
I20250115 14:52:14.984810 1168 raft_consensus.cc:357] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.985230 1168 raft_consensus.cc:383] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:14.985320 1168 raft_consensus.cc:738] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:14.985683 1168 consensus_queue.cc:260] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:14.986274 1168 ts_tablet_manager.cc:1428] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:14.986577 1168 tablet_bootstrap.cc:492] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:14.991310 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8650571f63d64554ae4cd30212f89d9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:14.992141 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8650571f63d64554ae4cd30212f89d9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.001858 1162 tablet_bootstrap.cc:492] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
W20250115 14:52:15.002810 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c6e6ffce5fa42f5a82f9b94982127e4 consensus state has no leader
I20250115 14:52:15.003690 1162 ts_tablet_manager.cc:1397] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.213s user 0.010s sys 0.008s
I20250115 14:52:15.004424 1162 raft_consensus.cc:357] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.004606 1162 raft_consensus.cc:383] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.004658 1162 raft_consensus.cc:738] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.004840 1162 consensus_queue.cc:260] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.005623 1162 ts_tablet_manager.cc:1428] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:15.005878 1162 tablet_bootstrap.cc:492] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:15.007655 1162 tablet_bootstrap.cc:654] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.009402 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8da9bd5ec841ebbc61b0f8de5ace3b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.010003 2403 raft_consensus.cc:491] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.010128 2403 raft_consensus.cc:513] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:15.010303 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c9a5593068d54d7bb3874cf434505e73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a5593068d54d7bb3874cf434505e73 already in progress: creating tablet
W20250115 14:52:15.010509 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5b33c876fab1476aba3b1f6e799fa96c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b33c876fab1476aba3b1f6e799fa96c already in progress: creating tablet
W20250115 14:52:15.010686 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet db57f7c3a4c14cc19d3a0624e5b1817a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db57f7c3a4c14cc19d3a0624e5b1817a already in progress: creating tablet
I20250115 14:52:15.010704 2403 leader_election.cc:290] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.010826 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet a48227ec5aa846c3ac63bd29ed38be25 (table default.Table_bb07dc16172246278fff8624127ab8bd [id=924019d3b4e842c2b3f173b307885151]) successfully deleted
I20250115 14:52:15.010968 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 246720dbb4ed4216ac87fb4a3df79e2c (table default.Table_daaa2755fe9d4f54ae6ff33ae452ec05 [id=8c0229e6f6884ea7be3417678bb27377]). Expected version 1 got 0
I20250115 14:52:15.011144 710 leader_election.cc:304] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:15.011473 2403 raft_consensus.cc:2743] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.012653 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df0250d08214a9783c392071126ed4a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:15.012861 1048 raft_consensus.cc:2388] T 5df0250d08214a9783c392071126ed4a P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2d7ecece2ab49e1be7079216517d13b in current term 1: Already voted for candidate 73c86158cc2641f182ea50a06634592d in this term.
I20250115 14:52:15.013270 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df0250d08214a9783c392071126ed4a" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:15.014706 1168 tablet_bootstrap.cc:654] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.015458 710 leader_election.cc:304] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:15.015861 2403 raft_consensus.cc:3054] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:15.016043 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8da9bd5ec841ebbc61b0f8de5ace3b" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:15.016230 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 00844e77811b4cbfb7240176055e2966
I20250115 14:52:15.016502 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70425c25919b4d0b999450210e1cb33f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.017316 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 00844e77811b4cbfb7240176055e2966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:15.017820 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 00844e77811b4cbfb7240176055e2966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00844e77811b4cbfb7240176055e2966 already in progress: creating tablet
I20250115 14:52:15.018220 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 00844e77811b4cbfb7240176055e2966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.018559 2403 raft_consensus.cc:2743] T 5df0250d08214a9783c392071126ed4a P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.022492 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2b26b9b1caf748aba7d36a04fbba2a80
I20250115 14:52:15.023285 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b26b9b1caf748aba7d36a04fbba2a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.024389 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1020437ffddf49cc846fcd3311892b46
W20250115 14:52:15.024803 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2b26b9b1caf748aba7d36a04fbba2a80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b26b9b1caf748aba7d36a04fbba2a80 already in progress: creating tablet
I20250115 14:52:15.024986 2763 tablet.cc:1721] T a6f3f8cae6244968927b864e7ce97565 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:15.025656 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 1020437ffddf49cc846fcd3311892b46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:15.026206 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1020437ffddf49cc846fcd3311892b46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1020437ffddf49cc846fcd3311892b46 already in progress: creating tablet
I20250115 14:52:15.027993 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5cb47732109e46d68e5af02ed52cb6d1
I20250115 14:52:15.028759 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 70425c25919b4d0b999450210e1cb33f
I20250115 14:52:15.028718 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 1020437ffddf49cc846fcd3311892b46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.029882 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b26b9b1caf748aba7d36a04fbba2a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.030568 2755 raft_consensus.cc:491] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.030777 2755 raft_consensus.cc:513] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.031909 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246720dbb4ed4216ac87fb4a3df79e2c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.032025 453 catalog_manager.cc:5526] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:15.032423 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b (table default.Table_f634603ea42b4215b8ef3df8c446ada7 [id=174b82e1e06f4073a74a25aad8fc8045]). Expected version 2 got 0
W20250115 14:52:15.032919 984 leader_election.cc:343] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.034062 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_dbb09336b4004c35b3ee4880598b02d8" } new_table_name: "default.Table_ea38f407592f4f49ba8e442f96e06873" modify_external_catalogs: true
I20250115 14:52:15.034708 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_298f14ed9b9b41fc802a48c4f1bbe07f" } new_table_name: "default.Table_f4904239f07b44888c5b08324a52aa5b" modify_external_catalogs: true
I20250115 14:52:15.035159 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 70425c25919b4d0b999450210e1cb33f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.035624 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb47732109e46d68e5af02ed52cb6d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.033128 2755 leader_election.cc:290] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.040544 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ab72fac2354c4767b21a84c1a2fc5dec" } new_table_name: "default.Table_a2cfae1230b24b128ccb10e1214ab247" modify_external_catalogs: true
I20250115 14:52:15.041471 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb47732109e46d68e5af02ed52cb6d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.043440 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 70425c25919b4d0b999450210e1cb33f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.053679 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_be4f13e702dc4d6cb81ab23dcfe5203e" } new_table_name: "default.Table_7bb0ca02228940ee91c77404cb564a29" modify_external_catalogs: true
I20250115 14:52:15.054105 1751 ts_tablet_manager.cc:1918] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:15.054209 1751 log.cc:1198] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ed777d7496514ec5986a7e6f53d532b1
I20250115 14:52:15.054584 1751 ts_tablet_manager.cc:1939] T ed777d7496514ec5986a7e6f53d532b1 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:15.032527 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246720dbb4ed4216ac87fb4a3df79e2c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:15.056521 911 raft_consensus.cc:2463] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:15.056972 983 leader_election.cc:304] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.036865 2549 raft_consensus.cc:491] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.057476 2549 raft_consensus.cc:513] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.058069 2549 leader_election.cc:290] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:15.058785 2755 raft_consensus.cc:2798] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:15.058915 2755 raft_consensus.cc:491] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:15.059024 2755 raft_consensus.cc:3054] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:15.059585 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7423aeb75aaf4a9f8d98517868158323"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.060163 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7423aeb75aaf4a9f8d98517868158323 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.061597 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
I20250115 14:52:15.061825 448 catalog_manager.cc:5526] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:15.062235 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet bf648bb656634def8189fb514a07e144 (table default.Table_fd84fd3d47974e05a580fae352838c20 [id=a00115ecf6954f83b3118654f98e629d]). Expected version 1 got 0
I20250115 14:52:15.062079 2755 raft_consensus.cc:513] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:15.039976 848 leader_election.cc:343] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.065057 848 leader_election.cc:304] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.065343 2549 raft_consensus.cc:2743] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:15.065816 848 consensus_peers.cc:487] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:15.066036 848 leader_election.cc:343] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.066185 848 leader_election.cc:304] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.066411 2549 raft_consensus.cc:2743] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:15.067167 848 leader_election.cc:343] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:15.067998 848 leader_election.cc:343] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:15.068336 848 leader_election.cc:343] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.053591 2770 tablet.cc:1721] T ac4d850b85e948939008e5275d42e3cc P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:52:15.069566 847 leader_election.cc:343] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e244689cef6844e2935becd265eecad0
W20250115 14:52:15.071256 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cda6870240d24eedaf9ddf57794b880e consensus state has no leader
W20250115 14:52:15.071555 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 16a88475f9ff42c1952e5a14ec852e90 consensus state has no leader
W20250115 14:52:15.071681 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5ab056252b5435693e0d5c041c9e1dc consensus state has no leader
W20250115 14:52:15.072556 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet edff9d5a740d4d5d8df02f4549c30700 has no consensus state
I20250115 14:52:15.063964 1011 tablet_service.cc:1467] Processing CreateTablet for tablet f869a6433f714f1999b41063fdd6b4ec (DEFAULT_TABLE table=default.Table_65c87128c01649dea82c71db78a50391 [id=f72a92afd99d448fb7f76dca8d47f4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.064611 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 2d660784270e452895ef25859f2004e2 (DEFAULT_TABLE table=default.Table_a829fe34660949f4808429fc815c1c11 [id=2e098d066e174a4788a3a44b1a18c3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.063102 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 3d951865a56c49d281312ec2f5b5bf19 (DEFAULT_TABLE table=default.Table_68412e63ed5047c9899420315f0314b9 [id=b5a851b29a8d4298b93eca7291ae1151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.075102 1164 tablet_bootstrap.cc:492] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.075268 1164 ts_tablet_manager.cc:1397] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.286s user 0.002s sys 0.004s
I20250115 14:52:15.074326 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f869a6433f714f1999b41063fdd6b4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.076071 1164 raft_consensus.cc:357] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.076265 1164 raft_consensus.cc:383] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.076344 1164 raft_consensus.cc:738] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.076604 1164 consensus_queue.cc:260] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.077081 1164 ts_tablet_manager.cc:1428] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:15.077335 1164 tablet_bootstrap.cc:492] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:15.039563 2631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250115 14:52:15.077787 2631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250115 14:52:15.077947 2631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250115 14:52:15.080058 773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70425c25919b4d0b999450210e1cb33f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:15.080163 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d660784270e452895ef25859f2004e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.080718 750 tablet_service.cc:1467] Processing CreateTablet for tablet 3d951865a56c49d281312ec2f5b5bf19 (DEFAULT_TABLE table=default.Table_68412e63ed5047c9899420315f0314b9 [id=b5a851b29a8d4298b93eca7291ae1151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.081045 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d951865a56c49d281312ec2f5b5bf19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.084362 2771 tablet.cc:1721] T ac4d850b85e948939008e5275d42e3cc P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:15.084444 738 tablet_service.cc:1467] Processing CreateTablet for tablet f869a6433f714f1999b41063fdd6b4ec (DEFAULT_TABLE table=default.Table_65c87128c01649dea82c71db78a50391 [id=f72a92afd99d448fb7f76dca8d47f4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.084743 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f869a6433f714f1999b41063fdd6b4ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:15.086717 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5cb47732109e46d68e5af02ed52cb6d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cb47732109e46d68e5af02ed52cb6d1 already in progress: creating tablet
W20250115 14:52:15.086982 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 70425c25919b4d0b999450210e1cb33f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70425c25919b4d0b999450210e1cb33f already in progress: creating tablet
I20250115 14:52:15.088521 747 tablet_service.cc:1467] Processing CreateTablet for tablet 2d660784270e452895ef25859f2004e2 (DEFAULT_TABLE table=default.Table_a829fe34660949f4808429fc815c1c11 [id=2e098d066e174a4788a3a44b1a18c3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.088842 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d660784270e452895ef25859f2004e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.089085 453 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 5df0250d08214a9783c392071126ed4a (table default.Table_90c834d5ef634e6aa797dd220e3d5faa [id=8ffbe34c2a11417f8ca50dbdd627bf22]). Expected version 2 got 0
I20250115 14:52:15.089655 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet 86c04fa23e7c40a5b5e07852e5d74a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.089882 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d951865a56c49d281312ec2f5b5bf19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.092511 740 tablet_service.cc:1467] Processing CreateTablet for tablet 806bf29f5dca4c55bdb14193e8fa2a82 (DEFAULT_TABLE table=default.Table_254b5a8c44db4a788415c8b69471bf9b [id=3bbd5fdd67be4496a204c90edd50545f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.092815 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 806bf29f5dca4c55bdb14193e8fa2a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.092693 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9be6523788f347129832e1de3230c4c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:15.092875 1164 tablet_bootstrap.cc:654] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:15.093207 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9be6523788f347129832e1de3230c4c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.093727 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 806bf29f5dca4c55bdb14193e8fa2a82 (DEFAULT_TABLE table=default.Table_254b5a8c44db4a788415c8b69471bf9b [id=3bbd5fdd67be4496a204c90edd50545f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.094046 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 806bf29f5dca4c55bdb14193e8fa2a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.098340 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 9e509a56f74749158aefec0054dc956d (DEFAULT_TABLE table=default.Table_e2b33135a54f436a95b73298d2b17a9e [id=991e7ddede9e40739a8f581e67c79151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.098646 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e509a56f74749158aefec0054dc956d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.105973 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d4ddf50a4f4844008070d3e57a94e88f (DEFAULT_TABLE table=default.Table_da1665493268478cbf1a922a31e0af9f [id=f5bc1c8875584fed89ae64daf75e319a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.106597 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ddf50a4f4844008070d3e57a94e88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.110812 2563 raft_consensus.cc:491] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.111090 2563 raft_consensus.cc:513] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.112406 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03eb225fad7459a955a3ec5e0002be1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.112525 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03eb225fad7459a955a3ec5e0002be1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:15.112893 984 leader_election.cc:343] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:15.113036 983 leader_election.cc:343] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.113133 983 leader_election.cc:304] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.113471 2563 leader_election.cc:290] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.113876 2563 raft_consensus.cc:2743] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.111864 2771 tablet.cc:1721] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:15.115312 2549 consensus_queue.cc:1035] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:15.120862 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 74329212a57348d6b09015d271774122 (DEFAULT_TABLE table=default.Table_32ddc413a4ee4d3fb9f1d439fe5cfc3b [id=934bd4df5e7a4db7816475edb830abe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.121397 1020 tablet_service.cc:1467] Processing CreateTablet for tablet be8dca820ffc45cdb190d80aa49948dd (DEFAULT_TABLE table=default.Table_8bfb65359dea4a2e810cb2827a3161a2 [id=9e5c940e3f6a4ddaa021c056f4b72738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.123680 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8dca820ffc45cdb190d80aa49948dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.124505 1164 tablet_bootstrap.cc:492] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.124624 1164 ts_tablet_manager.cc:1397] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.005s
I20250115 14:52:15.125336 1164 raft_consensus.cc:357] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.125511 1164 raft_consensus.cc:383] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.125551 1164 raft_consensus.cc:738] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.125747 1164 consensus_queue.cc:260] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.126231 1164 ts_tablet_manager.cc:1428] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250115 14:52:15.122326 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 96a5ce44bcb948f9abff0c20682e787f (DEFAULT_TABLE table=default.Table_0c704df3a19a436e891ea2813ceb7db3 [id=8b8dc63854714121ae44d22462a69d08]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:15.128362 2631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250115 14:52:15.128500 2631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250115 14:52:15.128572 2631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20250115 14:52:15.128619 1164 tablet_bootstrap.cc:492] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:15.128643 2631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250115 14:52:15.128645 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a5ce44bcb948f9abff0c20682e787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.130827 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_14bb1fc9753d4ab987aab9744d650965"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:15.130921 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_52240b9928824856bd6ce7d2ea06d625"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.131366 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52240b9928824856bd6ce7d2ea06d625 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:15.131366 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14bb1fc9753d4ab987aab9744d650965 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.138394 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74329212a57348d6b09015d271774122. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.140447 1164 tablet_bootstrap.cc:654] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.141858 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0e1d19fe704640f9a3cf90b94324f31b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:15.142094 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_793b55bcf9ea422588897663c768d387"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.142330 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e1d19fe704640f9a3cf90b94324f31b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:15.142416 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_793b55bcf9ea422588897663c768d387 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.143353 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 06aa85a23e9e435d9d0d91e807b47a81 (DEFAULT_TABLE table=default.Table_288d46f4b2dd44b3a5a4ab52188add56 [id=ba6fe4731e1f481f80696e7dda0d7d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.143476 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c0788d0ac02f4683aa546d7c0682b59d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:15.143672 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06aa85a23e9e435d9d0d91e807b47a81. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:15.143921 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0788d0ac02f4683aa546d7c0682b59d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.150081 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 0956531fdeb340208ad3ccab5e934a18 (DEFAULT_TABLE table=default.Table_442bce8b3c9842cfba4b7f94c280eb3e [id=e6eefa19b9d949d18862c4906140fe5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.150493 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0956531fdeb340208ad3ccab5e934a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.123191 1015 tablet_service.cc:1467] Processing CreateTablet for tablet e7cebc28a7f54ab6baf1d1505cb5fb0d (DEFAULT_TABLE table=default.Table_c62569d637274fd68dc1442fbe05f902 [id=4609b7dff56a49d69c8a2493f86b9466]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.150817 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff284a245a940fcb4567f95cf8fd424 (DEFAULT_TABLE table=default.Table_214a95027bd94905b10b5c7285a31b00 [id=e23ef49f13c943ea9fe6f1c2f6a0c9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.122812 1021 tablet_service.cc:1467] Processing CreateTablet for tablet ce693a5c338f4e7394e0f5c323b8942b (DEFAULT_TABLE table=default.Table_633028cf8b914cac8836453b1e242fdf [id=8b623002868449dbba9a0ccf51629a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.152859 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7cebc28a7f54ab6baf1d1505cb5fb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.159653 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce693a5c338f4e7394e0f5c323b8942b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.151247 1020 tablet_service.cc:1467] Processing CreateTablet for tablet 03892c1eaf5a474dbba9ad35cbf0ec4d (DEFAULT_TABLE table=default.Table_5c321ec5b2c84f01a09fa02678c31b4b [id=549b510c3dd14274900b29d4b70d24ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.161726 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03892c1eaf5a474dbba9ad35cbf0ec4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.162122 1024 tablet_service.cc:1467] Processing CreateTablet for tablet df7884f209d94e4ab6ebb31883bd8ee4 (DEFAULT_TABLE table=default.Table_b2b4a2f38c7444bd9638bc834e8bb2a1 [id=e63d1868be5c47608dde5880427cf821]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.162763 1019 tablet_service.cc:1467] Processing CreateTablet for tablet f5aa92f75f69448d9c43cc808c621f68 (DEFAULT_TABLE table=default.Table_a0070c4d1be947df84bf19c1efb88fa1 [id=84efaff9f0d44937a7379bf76905e838]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.162798 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8 (DEFAULT_TABLE table=default.Table_5b2238ba86fe41079543db9d13c4c1f4 [id=87305597cbdb42d595750887c01ea856]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.163838 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5aa92f75f69448d9c43cc808c621f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.164016 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 554990a441034b25afd6f7d74505e76d (DEFAULT_TABLE table=default.Table_46ab5287629b4006b72bb93ccde2b813 [id=d2548227f1fd4378a61bdc488374821a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.164332 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 554990a441034b25afd6f7d74505e76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.168512 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7884f209d94e4ab6ebb31883bd8ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.169636 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.172214 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff284a245a940fcb4567f95cf8fd424. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.175849 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 678c0a7256ce4bd19a84de83e2e8633c (DEFAULT_TABLE table=default.Table_5e15a5e365604ac2a0fc0ac6631a167f [id=e9c9601885f148d49cc405a65494f2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.176132 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 8c91f5a41ea3443e88a8aaae9aa42a18 (DEFAULT_TABLE table=default.Table_0b03e1670f0c48dab3a8f103e6bf6dbd [id=1eeea7ded7c8454cacec6e9802df0a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.178306 1018 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2ffb75f16c4244b795c21f8b89d32a (DEFAULT_TABLE table=default.Table_d384734a041a4ebf963744b6f24a2572 [id=64678b18e5bd42c28d03b195f3603d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.180418 750 tablet_service.cc:1467] Processing CreateTablet for tablet ce693a5c338f4e7394e0f5c323b8942b (DEFAULT_TABLE table=default.Table_633028cf8b914cac8836453b1e242fdf [id=8b623002868449dbba9a0ccf51629a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.180791 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce693a5c338f4e7394e0f5c323b8942b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.181354 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce36c783994c3badf985b79583c0a2 (DEFAULT_TABLE table=default.Table_54c4b2a5cfdc4d65b61e0bf4854a48b8 [id=fe667afae12f49bd93e7ef66aed0c1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.182266 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb0195969e44d8d9d770c8b0d9c85bf (DEFAULT_TABLE table=default.Table_15773744107143d6b1e5215c832f0a3f [id=f4305d1bfe12463fa68fb1405be5267f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.183180 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 063d0e08a8a949f28c0ac62b2e4ec346 (DEFAULT_TABLE table=default.Table_9ff76eab7cfc4904a16a9e1b4ff89486 [id=f4009f6c09084261a84edc3cfedef906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.183784 1023 tablet_service.cc:1467] Processing CreateTablet for tablet d64d0b02274c47b9ba0ce12dac486031 (DEFAULT_TABLE table=default.Table_e1c85cd6cf714918b357548b0bc26251 [id=233fdd3693334589b22b15dc145108ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.184523 738 tablet_service.cc:1467] Processing CreateTablet for tablet d4ddf50a4f4844008070d3e57a94e88f (DEFAULT_TABLE table=default.Table_da1665493268478cbf1a922a31e0af9f [id=f5bc1c8875584fed89ae64daf75e319a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.184849 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ddf50a4f4844008070d3e57a94e88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.185650 1022 tablet_service.cc:1467] Processing CreateTablet for tablet bd745bda7c614d72baab9ec7ccaa3239 (DEFAULT_TABLE table=default.Table_adabfc70f8eb4826ae67c16382a57ee9 [id=5ae85c348b9c42afb229c07c320291f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.188544 1026 tablet_service.cc:1467] Processing CreateTablet for tablet dfa26077522d4641b0e06cb06c6d8dcf (DEFAULT_TABLE table=default.Table_597d1c61c9bc4e398eaf813ad56c8841 [id=411906b36e384ab29c6e4fa709fa44a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.188795 747 tablet_service.cc:1467] Processing CreateTablet for tablet 74329212a57348d6b09015d271774122 (DEFAULT_TABLE table=default.Table_32ddc413a4ee4d3fb9f1d439fe5cfc3b [id=934bd4df5e7a4db7816475edb830abe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.189117 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74329212a57348d6b09015d271774122. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.189376 741 tablet_service.cc:1467] Processing CreateTablet for tablet be8dca820ffc45cdb190d80aa49948dd (DEFAULT_TABLE table=default.Table_8bfb65359dea4a2e810cb2827a3161a2 [id=9e5c940e3f6a4ddaa021c056f4b72738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.189904 1013 tablet_service.cc:1467] Processing CreateTablet for tablet eeb5ad85b4524994bba041dfb3fc8e1f (DEFAULT_TABLE table=default.Table_4980e328dcce4600a482147b7a9c0b30 [id=ce371a899b4543f6bdbfe5868d65d714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.190533 753 tablet_service.cc:1467] Processing CreateTablet for tablet e7cebc28a7f54ab6baf1d1505cb5fb0d (DEFAULT_TABLE table=default.Table_c62569d637274fd68dc1442fbe05f902 [id=4609b7dff56a49d69c8a2493f86b9466]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.190833 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7cebc28a7f54ab6baf1d1505cb5fb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.189663 751 tablet_service.cc:1467] Processing CreateTablet for tablet 96a5ce44bcb948f9abff0c20682e787f (DEFAULT_TABLE table=default.Table_0c704df3a19a436e891ea2813ceb7db3 [id=8b8dc63854714121ae44d22462a69d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.191458 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a5ce44bcb948f9abff0c20682e787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.192308 739 tablet_service.cc:1467] Processing CreateTablet for tablet 06aa85a23e9e435d9d0d91e807b47a81 (DEFAULT_TABLE table=default.Table_288d46f4b2dd44b3a5a4ab52188add56 [id=ba6fe4731e1f481f80696e7dda0d7d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.192904 755 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff284a245a940fcb4567f95cf8fd424 (DEFAULT_TABLE table=default.Table_214a95027bd94905b10b5c7285a31b00 [id=e23ef49f13c943ea9fe6f1c2f6a0c9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.188519 740 tablet_service.cc:1467] Processing CreateTablet for tablet 9e509a56f74749158aefec0054dc956d (DEFAULT_TABLE table=default.Table_e2b33135a54f436a95b73298d2b17a9e [id=991e7ddede9e40739a8f581e67c79151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.198035 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e509a56f74749158aefec0054dc956d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.200652 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff284a245a940fcb4567f95cf8fd424. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.202591 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8dca820ffc45cdb190d80aa49948dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.202800 1164 tablet_bootstrap.cc:492] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.202910 1164 ts_tablet_manager.cc:1397] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.013s
I20250115 14:52:15.205624 1024 tablet_service.cc:1467] Processing CreateTablet for tablet 6c68d7ac544c4606bacabc601f6675f7 (DEFAULT_TABLE table=default.Table_141efb598ef84dfdace28d40f3843d86 [id=6fb7f3061f0041e0a834ad48fdc1f8e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.205996 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c68d7ac544c4606bacabc601f6675f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.206559 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3e67bbd220754aac89a7b142edcb206c" } new_table_name: "default.Table_669581139cfa4cffa25ef15e5285c455" modify_external_catalogs: true
I20250115 14:52:15.208088 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4bdcbfbff073498cb6257975ee190669"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.208575 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bdcbfbff073498cb6257975ee190669 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.209084 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678c0a7256ce4bd19a84de83e2e8633c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.209262 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeb5ad85b4524994bba041dfb3fc8e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.209424 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb0195969e44d8d9d770c8b0d9c85bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.208088 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_69be76f118b247d7b98ab4ec45332da0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:15.209842 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64d0b02274c47b9ba0ce12dac486031. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:15.210017 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69be76f118b247d7b98ab4ec45332da0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.211089 2563 raft_consensus.cc:491] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.211210 2563 raft_consensus.cc:513] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.211757 2563 leader_election.cc:290] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.212422 1164 raft_consensus.cc:357] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.212594 1164 raft_consensus.cc:383] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.212641 1164 raft_consensus.cc:738] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.212857 1164 consensus_queue.cc:260] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.213356 1164 ts_tablet_manager.cc:1428] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250115 14:52:15.213634 1164 tablet_bootstrap.cc:492] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:15.215981 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 16a88475f9ff42c1952e5a14ec852e90 (table default.Table_f4904239f07b44888c5b08324a52aa5b [id=3c1f2f64fe374b5294ef6130f252840f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 16a88475f9ff42c1952e5a14ec852e90 consensus state has no leader
I20250115 14:52:15.218168 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_08ec29c2d0ca4d7db6cd48a97784dced" } modify_external_catalogs: true
I20250115 14:52:15.218322 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_08ec29c2d0ca4d7db6cd48a97784dced" } modify_external_catalogs: true
I20250115 14:52:15.219537 1164 tablet_bootstrap.cc:654] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:15.219547 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1218f0bdb1e14bd78d76c6a06805f38c consensus state has no leader
I20250115 14:52:15.224637 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0645565e1b904cff8d1ed879748d01bd" } modify_external_catalogs: true
I20250115 14:52:15.224831 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0645565e1b904cff8d1ed879748d01bd" } modify_external_catalogs: true
I20250115 14:52:15.226231 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_262d362a9eb447c689cbb66e877ab7be" } modify_external_catalogs: true
I20250115 14:52:15.226370 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_262d362a9eb447c689cbb66e877ab7be" } modify_external_catalogs: true
I20250115 14:52:15.231148 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_478cc36362494028a43f0e71465e1834" } modify_external_catalogs: true
I20250115 14:52:15.231196 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_683d27a9444245be9ff47c13b148e508" } modify_external_catalogs: true
I20250115 14:52:15.231315 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_478cc36362494028a43f0e71465e1834" } modify_external_catalogs: true
I20250115 14:52:15.231389 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_683d27a9444245be9ff47c13b148e508" } modify_external_catalogs: true
W20250115 14:52:15.232362 984 consensus_peers.cc:487] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f06544890cbf460cbe218690e998eb1d. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:15.236871 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1259256b7744e739581125611abbe67" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:15.238140 984 leader_election.cc:343] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.239478 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4ee8b63cc21145d8993b058d8c6d23c7
I20250115 14:52:15.240975 755 tablet_service.cc:1467] Processing CreateTablet for tablet 0956531fdeb340208ad3ccab5e934a18 (DEFAULT_TABLE table=default.Table_442bce8b3c9842cfba4b7f94c280eb3e [id=e6eefa19b9d949d18862c4906140fe5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.241400 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0956531fdeb340208ad3ccab5e934a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.250066 740 tablet_service.cc:1467] Processing CreateTablet for tablet 03892c1eaf5a474dbba9ad35cbf0ec4d (DEFAULT_TABLE table=default.Table_5c321ec5b2c84f01a09fa02678c31b4b [id=549b510c3dd14274900b29d4b70d24ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.250499 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03892c1eaf5a474dbba9ad35cbf0ec4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.251518 2631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35583,127.0.104.61:45653,127.0.104.60:40227
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:40227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=37445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 75b75ca2f58f1a5aba69207830e3558011033d99
build type FASTDEBUG
built by None at 15 Jan 2025 14:43:22 UTC on 24a791456cd2
build id 4191
ASAN enabled
I20250115 14:52:15.254169 2631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250115 14:52:15.256443 2631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250115 14:52:15.266876 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_139b3230a751497792db52e42d6fcd5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.267465 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_139b3230a751497792db52e42d6fcd5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.270287 1013 tablet_service.cc:1467] Processing CreateTablet for tablet a3c4c683d7c040bcaa13ff73f9f16613 (DEFAULT_TABLE table=default.Table_84014c06851e40abab7729b00a5aac85 [id=fa8760aa8b62455e942ab02135a67c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.275197 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_da940b0d8b7d4e409e73adab5c24ba76" } new_table_name: "default.Table_74825cfc55d54eab828b4eef26d5148e" modify_external_catalogs: true
I20250115 14:52:15.275321 1019 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ee8b63cc21145d8993b058d8c6d23c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.275417 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2e2abefaccf54e469bacf8126edad0c4" } new_table_name: "default.Table_1365cebf9380460289484fcc7b729f58" modify_external_catalogs: true
I20250115 14:52:15.276631 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_233462856fc84da29a45b15193b81652" } new_table_name: "default.Table_a3ebaf4214574dd68c619382cb749409" modify_external_catalogs: true
I20250115 14:52:15.277658 450 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8650571f63d64554ae4cd30212f89d9b" } new_table_name: "default.Table_cf227e000a9a4a8f8565e985bb801c61" modify_external_catalogs: true
I20250115 14:52:15.278718 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_892d26a61a5242e293e5fb8cff0d798c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.279152 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_892d26a61a5242e293e5fb8cff0d798c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:15.286049 983 consensus_peers.cc:487] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 -> Peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Couldn't send request to peer 73c86158cc2641f182ea50a06634592d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:15.069772 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1192) took 1543 ms. Trace:
I20250115 14:52:15.287257 747 tablet_service.cc:1467] Processing CreateTablet for tablet df7884f209d94e4ab6ebb31883bd8ee4 (DEFAULT_TABLE table=default.Table_b2b4a2f38c7444bd9638bc834e8bb2a1 [id=e63d1868be5c47608dde5880427cf821]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.287647 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7884f209d94e4ab6ebb31883bd8ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.285193 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1259256b7744e739581125611abbe67" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:15.290735 983 leader_election.cc:343] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.290855 983 leader_election.cc:304] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.291205 2563 raft_consensus.cc:2743] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.293473 2549 consensus_queue.cc:1035] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:15.300719 2773 tablet.cc:1721] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250115 14:52:15.304562 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 consensus state has no leader
W20250115 14:52:15.304858 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 87b28e83081a4f6e959c8b26c68b81ce consensus state has no leader
I20250115 14:52:15.304975 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c3229d9207b641aaacbf3a4544824974"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.305229 450 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c6eacb0c16f4c4ca8db8eb8f65be1ae has no consensus state
W20250115 14:52:15.305455 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3229d9207b641aaacbf3a4544824974 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.326411 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246720dbb4ed4216ac87fb4a3df79e2c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:15.326972 984 leader_election.cc:343] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.330667 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 00844e77811b4cbfb7240176055e2966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:15.331136 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b26b9b1caf748aba7d36a04fbba2a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:15.331496 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 1020437ffddf49cc846fcd3311892b46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:15.331883 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb47732109e46d68e5af02ed52cb6d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:15.332341 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 70425c25919b4d0b999450210e1cb33f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:15.333034 879 tablet_service.cc:1467] Processing CreateTablet for tablet 3d951865a56c49d281312ec2f5b5bf19 (DEFAULT_TABLE table=default.Table_68412e63ed5047c9899420315f0314b9 [id=b5a851b29a8d4298b93eca7291ae1151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.333410 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d951865a56c49d281312ec2f5b5bf19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.334319 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e244689cef6844e2935becd265eecad0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1a69b6de5d433f82579b1259548fd2) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:15.334909 891 tablet_service.cc:1467] Processing CreateTablet for tablet 806bf29f5dca4c55bdb14193e8fa2a82 (DEFAULT_TABLE table=default.Table_254b5a8c44db4a788415c8b69471bf9b [id=3bbd5fdd67be4496a204c90edd50545f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.335234 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 806bf29f5dca4c55bdb14193e8fa2a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.336519 884 tablet_service.cc:1467] Processing CreateTablet for tablet f869a6433f714f1999b41063fdd6b4ec (DEFAULT_TABLE table=default.Table_65c87128c01649dea82c71db78a50391 [id=f72a92afd99d448fb7f76dca8d47f4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.337028 882 tablet_service.cc:1467] Processing CreateTablet for tablet 2d660784270e452895ef25859f2004e2 (DEFAULT_TABLE table=default.Table_a829fe34660949f4808429fc815c1c11 [id=2e098d066e174a4788a3a44b1a18c3a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.337459 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f869a6433f714f1999b41063fdd6b4ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.338943 881 tablet_service.cc:1467] Processing CreateTablet for tablet ce693a5c338f4e7394e0f5c323b8942b (DEFAULT_TABLE table=default.Table_633028cf8b914cac8836453b1e242fdf [id=8b623002868449dbba9a0ccf51629a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.340386 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d660784270e452895ef25859f2004e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.341420 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce693a5c338f4e7394e0f5c323b8942b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.344934 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d328c98aff614867b35565e6d7aecad8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.345475 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d328c98aff614867b35565e6d7aecad8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.348189 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 5130494a392144e7befd2183a015736b (DEFAULT_TABLE table=default.Table_37389ea3789646978b1da3e473f46631 [id=be258192053d401fb9cb497ddb36debc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.349200 2755 leader_election.cc:290] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.349725 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "246720dbb4ed4216ac87fb4a3df79e2c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:15.349903 906 raft_consensus.cc:3054] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:15.288352 1751 rpcz_store.cc:276] 0115 14:52:13.526265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.526315 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:14.941974 (+1415659us) ts_tablet_manager.cc:1145] Deleting tablet ed777d7496514ec5986a7e6f53d532b1
0115 14:52:14.941983 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:14.972786 (+ 30803us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:15.053995 (+ 81209us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:15.069753 (+ 15758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1390164}
I20250115 14:52:15.358083 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ed777d7496514ec5986a7e6f53d532b1 (table default.Table_56da29ad1fe24546bc5c6a4a17a774e7 [id=1b1cf9caa0194880af26c65d6f7bcbf5]) successfully deleted
I20250115 14:52:15.358263 906 raft_consensus.cc:2463] T 246720dbb4ed4216ac87fb4a3df79e2c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:15.358505 1751 tablet_replica.cc:331] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:15.358706 983 leader_election.cc:304] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.365388 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8e77605904c9425cae44eed3f031d5f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.365979 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e77605904c9425cae44eed3f031d5f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.368566 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_230e0c86726c489cb19444c0ebea8a9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.369053 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_230e0c86726c489cb19444c0ebea8a9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.371327 2563 raft_consensus.cc:2798] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:15.371451 2563 raft_consensus.cc:695] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:15.371677 2563 consensus_queue.cc:237] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.372840 2758 raft_consensus.cc:491] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.372965 2758 raft_consensus.cc:513] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.358753 1751 raft_consensus.cc:2238] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:15.376906 1751 raft_consensus.cc:2267] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:15.377444 1751 ts_tablet_manager.cc:1905] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:15.391273 2758 leader_election.cc:290] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.392563 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560d42c15c64b4c9c0cfe95706b4d9e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:15.393210 983 leader_election.cc:343] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.395560 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560d42c15c64b4c9c0cfe95706b4d9e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:15.395998 984 leader_election.cc:343] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.396112 984 leader_election.cc:304] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.396572 2758 raft_consensus.cc:2743] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.403302 747 tablet_service.cc:1467] Processing CreateTablet for tablet f5aa92f75f69448d9c43cc808c621f68 (DEFAULT_TABLE table=default.Table_a0070c4d1be947df84bf19c1efb88fa1 [id=84efaff9f0d44937a7379bf76905e838]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.406508 1162 tablet_bootstrap.cc:492] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:15.407126 1162 ts_tablet_manager.cc:1397] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.401s user 0.019s sys 0.029s
I20250115 14:52:15.417304 1164 tablet_bootstrap.cc:492] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.417447 1164 ts_tablet_manager.cc:1397] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.204s user 0.003s sys 0.012s
I20250115 14:52:15.418768 750 tablet_service.cc:1467] Processing CreateTablet for tablet 5130494a392144e7befd2183a015736b (DEFAULT_TABLE table=default.Table_37389ea3789646978b1da3e473f46631 [id=be258192053d401fb9cb497ddb36debc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.418964 753 tablet_service.cc:1467] Processing CreateTablet for tablet 554990a441034b25afd6f7d74505e76d (DEFAULT_TABLE table=default.Table_46ab5287629b4006b72bb93ccde2b813 [id=d2548227f1fd4378a61bdc488374821a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.419212 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5130494a392144e7befd2183a015736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.419389 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 554990a441034b25afd6f7d74505e76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.419502 741 tablet_service.cc:1467] Processing CreateTablet for tablet 8c91f5a41ea3443e88a8aaae9aa42a18 (DEFAULT_TABLE table=default.Table_0b03e1670f0c48dab3a8f103e6bf6dbd [id=1eeea7ded7c8454cacec6e9802df0a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.419778 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c91f5a41ea3443e88a8aaae9aa42a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.420864 751 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2ffb75f16c4244b795c21f8b89d32a (DEFAULT_TABLE table=default.Table_d384734a041a4ebf963744b6f24a2572 [id=64678b18e5bd42c28d03b195f3603d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.421155 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2ffb75f16c4244b795c21f8b89d32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.400895 1167 ts_tablet_manager.cc:1918] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:15.421784 1167 log.cc:1198] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/f06544890cbf460cbe218690e998eb1d
I20250115 14:52:15.422122 1167 ts_tablet_manager.cc:1939] T f06544890cbf460cbe218690e998eb1d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:15.422401 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1208) took 1440 ms. Trace:
I20250115 14:52:15.423326 755 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8 (DEFAULT_TABLE table=default.Table_5b2238ba86fe41079543db9d13c4c1f4 [id=87305597cbdb42d595750887c01ea856]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.423599 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.424193 756 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb0195969e44d8d9d770c8b0d9c85bf (DEFAULT_TABLE table=default.Table_15773744107143d6b1e5215c832f0a3f [id=f4305d1bfe12463fa68fb1405be5267f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.425313 1162 raft_consensus.cc:357] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.425627 1162 raft_consensus.cc:383] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.425781 1162 raft_consensus.cc:738] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.426170 1162 consensus_queue.cc:260] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.427666 888 tablet_service.cc:1467] Processing CreateTablet for tablet d4ddf50a4f4844008070d3e57a94e88f (DEFAULT_TABLE table=default.Table_da1665493268478cbf1a922a31e0af9f [id=f5bc1c8875584fed89ae64daf75e319a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.427917 884 tablet_service.cc:1467] Processing CreateTablet for tablet 74329212a57348d6b09015d271774122 (DEFAULT_TABLE table=default.Table_32ddc413a4ee4d3fb9f1d439fe5cfc3b [id=934bd4df5e7a4db7816475edb830abe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.428026 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4ddf50a4f4844008070d3e57a94e88f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.428237 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74329212a57348d6b09015d271774122. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.422469 1167 rpcz_store.cc:276] 0115 14:52:13.982111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.982163 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:14.901825 (+919662us) ts_tablet_manager.cc:1145] Deleting tablet f06544890cbf460cbe218690e998eb1d
0115 14:52:14.901831 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.049598 (+147767us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:15.400760 (+351162us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:15.422386 (+ 21626us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10067,"tablet-delete.queue_time_us":919492}
I20250115 14:52:15.431380 891 tablet_service.cc:1467] Processing CreateTablet for tablet 9e509a56f74749158aefec0054dc956d (DEFAULT_TABLE table=default.Table_e2b33135a54f436a95b73298d2b17a9e [id=991e7ddede9e40739a8f581e67c79151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.426968 1162 ts_tablet_manager.cc:1428] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20250115 14:52:15.438051 879 tablet_service.cc:1467] Processing CreateTablet for tablet be8dca820ffc45cdb190d80aa49948dd (DEFAULT_TABLE table=default.Table_8bfb65359dea4a2e810cb2827a3161a2 [id=9e5c940e3f6a4ddaa021c056f4b72738]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.438555 882 tablet_service.cc:1467] Processing CreateTablet for tablet 96a5ce44bcb948f9abff0c20682e787f (DEFAULT_TABLE table=default.Table_0c704df3a19a436e891ea2813ceb7db3 [id=8b8dc63854714121ae44d22462a69d08]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.437338 1162 tablet_bootstrap.cc:492] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:15.439056 881 tablet_service.cc:1467] Processing CreateTablet for tablet e7cebc28a7f54ab6baf1d1505cb5fb0d (DEFAULT_TABLE table=default.Table_c62569d637274fd68dc1442fbe05f902 [id=4609b7dff56a49d69c8a2493f86b9466]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.444684 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96a5ce44bcb948f9abff0c20682e787f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.444936 738 tablet_service.cc:1467] Processing CreateTablet for tablet 678c0a7256ce4bd19a84de83e2e8633c (DEFAULT_TABLE table=default.Table_5e15a5e365604ac2a0fc0ac6631a167f [id=e9c9601885f148d49cc405a65494f2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.445355 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678c0a7256ce4bd19a84de83e2e8633c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.448614 754 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce36c783994c3badf985b79583c0a2 (DEFAULT_TABLE table=default.Table_54c4b2a5cfdc4d65b61e0bf4854a48b8 [id=fe667afae12f49bd93e7ef66aed0c1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.448976 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ce36c783994c3badf985b79583c0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.449235 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb0195969e44d8d9d770c8b0d9c85bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.450500 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e509a56f74749158aefec0054dc956d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.452610 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7cebc28a7f54ab6baf1d1505cb5fb0d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:15.453714 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 70989.9us
I20250115 14:52:15.455276 883 tablet_service.cc:1467] Processing CreateTablet for tablet 06aa85a23e9e435d9d0d91e807b47a81 (DEFAULT_TABLE table=default.Table_288d46f4b2dd44b3a5a4ab52188add56 [id=ba6fe4731e1f481f80696e7dda0d7d6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.455590 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06aa85a23e9e435d9d0d91e807b47a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.455606 893 tablet_service.cc:1467] Processing CreateTablet for tablet df7884f209d94e4ab6ebb31883bd8ee4 (DEFAULT_TABLE table=default.Table_b2b4a2f38c7444bd9638bc834e8bb2a1 [id=e63d1868be5c47608dde5880427cf821]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.455896 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7884f209d94e4ab6ebb31883bd8ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.456501 892 tablet_service.cc:1467] Processing CreateTablet for tablet 03892c1eaf5a474dbba9ad35cbf0ec4d (DEFAULT_TABLE table=default.Table_5c321ec5b2c84f01a09fa02678c31b4b [id=549b510c3dd14274900b29d4b70d24ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.456776 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03892c1eaf5a474dbba9ad35cbf0ec4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.460512 878 tablet_service.cc:1467] Processing CreateTablet for tablet 0956531fdeb340208ad3ccab5e934a18 (DEFAULT_TABLE table=default.Table_442bce8b3c9842cfba4b7f94c280eb3e [id=e6eefa19b9d949d18862c4906140fe5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.460793 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0956531fdeb340208ad3ccab5e934a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.462049 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1209) took 1455 ms. Trace:
I20250115 14:52:15.462122 1167 rpcz_store.cc:276] 0115 14:52:14.007033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.007142 (+ 109us) service_pool.cc:225] Handling call
0115 14:52:15.432741 (+1425599us) ts_tablet_manager.cc:1145] Deleting tablet f2993e1c95234a1db2ae98a8e08c43f8
0115 14:52:15.443805 (+ 11064us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.462030 (+ 18225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1425131}
I20250115 14:52:15.462386 1167 tablet_replica.cc:331] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:15.462558 1167 raft_consensus.cc:2238] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:15.456506 885 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff284a245a940fcb4567f95cf8fd424 (DEFAULT_TABLE table=default.Table_214a95027bd94905b10b5c7285a31b00 [id=e23ef49f13c943ea9fe6f1c2f6a0c9d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.466054 883 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8 (DEFAULT_TABLE table=default.Table_5b2238ba86fe41079543db9d13c4c1f4 [id=87305597cbdb42d595750887c01ea856]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.464604 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be8dca820ffc45cdb190d80aa49948dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.466415 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb5fd4a7fd64f188b2ba3fbe2881ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.466570 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ff284a245a940fcb4567f95cf8fd424. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.469122 1168 tablet_bootstrap.cc:492] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:15.469244 1168 ts_tablet_manager.cc:1397] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.483s user 0.004s sys 0.007s
I20250115 14:52:15.469867 1168 raft_consensus.cc:357] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.470059 1168 raft_consensus.cc:383] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.470106 1168 raft_consensus.cc:738] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.470276 1168 consensus_queue.cc:260] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.470855 1168 ts_tablet_manager.cc:1428] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:15.471103 1168 tablet_bootstrap.cc:492] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:15.471665 2563 raft_consensus.cc:491] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.471804 2563 raft_consensus.cc:513] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.472498 1168 tablet_bootstrap.cc:654] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.474449 1167 raft_consensus.cc:2267] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:15.474685 2770 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:15.477998 737 tablet_service.cc:1467] Processing CreateTablet for tablet 063d0e08a8a949f28c0ac62b2e4ec346 (DEFAULT_TABLE table=default.Table_9ff76eab7cfc4904a16a9e1b4ff89486 [id=f4009f6c09084261a84edc3cfedef906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.478147 749 tablet_service.cc:1467] Processing CreateTablet for tablet dfa26077522d4641b0e06cb06c6d8dcf (DEFAULT_TABLE table=default.Table_597d1c61c9bc4e398eaf813ad56c8841 [id=411906b36e384ab29c6e4fa709fa44a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.478382 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063d0e08a8a949f28c0ac62b2e4ec346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.478524 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ee8b63cc21145d8993b058d8c6d23c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.478736 1166 tablet_replica.cc:331] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:15.478991 1166 raft_consensus.cc:2238] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:15.479328 1166 raft_consensus.cc:2267] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:15.478768 748 tablet_service.cc:1467] Processing CreateTablet for tablet bd745bda7c614d72baab9ec7ccaa3239 (DEFAULT_TABLE table=default.Table_adabfc70f8eb4826ae67c16382a57ee9 [id=5ae85c348b9c42afb229c07c320291f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.479866 1166 ts_tablet_manager.cc:1905] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:15.478734 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa26077522d4641b0e06cb06c6d8dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.481813 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6978223c1984a38bec5f2d06a9cf457" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:15.482286 893 tablet_service.cc:1467] Processing CreateTablet for tablet 5130494a392144e7befd2183a015736b (DEFAULT_TABLE table=default.Table_37389ea3789646978b1da3e473f46631 [id=be258192053d401fb9cb497ddb36debc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:15.482329 983 leader_election.cc:343] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.482626 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5130494a392144e7befd2183a015736b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:15.486161 2788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:52:15.487540 1168 tablet_bootstrap.cc:492] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:15.487732 1168 ts_tablet_manager.cc:1397] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250115 14:52:15.488597 1168 raft_consensus.cc:357] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.488911 1168 raft_consensus.cc:383] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.489009 1168 raft_consensus.cc:738] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.489293 1168 consensus_queue.cc:260] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:15.489812 2793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:52:15.490196 1168 ts_tablet_manager.cc:1428] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:15.491135 1168 tablet_bootstrap.cc:492] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:15.481215 2563 leader_election.cc:290] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.493664 1168 tablet_bootstrap.cc:654] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.495083 745 tablet_service.cc:1467] Processing CreateTablet for tablet 6c68d7ac544c4606bacabc601f6675f7 (DEFAULT_TABLE table=default.Table_141efb598ef84dfdace28d40f3843d86 [id=6fb7f3061f0041e0a834ad48fdc1f8e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.495139 742 tablet_service.cc:1467] Processing CreateTablet for tablet eeb5ad85b4524994bba041dfb3fc8e1f (DEFAULT_TABLE table=default.Table_4980e328dcce4600a482147b7a9c0b30 [id=ce371a899b4543f6bdbfe5868d65d714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.495308 2631 server_base.cc:1034] running on GCE node
I20250115 14:52:15.494638 743 tablet_service.cc:1467] Processing CreateTablet for tablet d64d0b02274c47b9ba0ce12dac486031 (DEFAULT_TABLE table=default.Table_e1c85cd6cf714918b357548b0bc26251 [id=233fdd3693334589b22b15dc145108ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.496994 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e244689cef6844e2935becd265eecad0" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.498312 2774 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:15.511550 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ae26bf8b34aa4bd2a498fb5959363aa5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:15.511963 744 tablet_service.cc:1467] Processing CreateTablet for tablet a3c4c683d7c040bcaa13ff73f9f16613 (DEFAULT_TABLE table=default.Table_84014c06851e40abab7729b00a5aac85 [id=fa8760aa8b62455e942ab02135a67c47]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:15.512096 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae26bf8b34aa4bd2a498fb5959363aa5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.514005 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42]). Expected version 3 got 0
I20250115 14:52:15.514186 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet ad988e25800f4af9a069b5a80582f749 (table default.Table_162eb94aa1674d21ae644021401a54e8 [id=e3887aa962874cfaa7ee89503ca1a49a]). Expected version 1 got 0
I20250115 14:52:15.516821 1020 tablet_service.cc:1467] Processing CreateTablet for tablet cb6c5005cfec426f94e401cfee6cf1c1 (DEFAULT_TABLE table=default.Table_252a71de31274362a56b0a5b528f14e9 [id=d89b158670e6490b97c1d0be25662838]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.516793 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e5b03e341bbd4d11af2f2a999e4ffbcc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.517261 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5b03e341bbd4d11af2f2a999e4ffbcc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.518549 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d384734a041a4ebf963744b6f24a2572" } new_table_name: "default.Table_e29cb0a0a76046cab39de4bf04512200" modify_external_catalogs: true
W20250115 14:52:15.518646 451 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd (table default.Table_1213cde4dcb6474487bba376e4516b9c [id=017270e9a3ad41f18c34b3cfc2c4cd42], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 1dbc2ae77d974b80bc1c93a3e44f3dcd consensus state has no leader
I20250115 14:52:15.519086 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1dbc2a3b624144dcaf2ea044013159bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.519469 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1dbc2a3b624144dcaf2ea044013159bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.519846 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_714851f5d34f456093fb3c60acb57e94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.520218 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_714851f5d34f456093fb3c60acb57e94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.520937 746 tablet_service.cc:1467] Processing CreateTablet for tablet cb6c5005cfec426f94e401cfee6cf1c1 (DEFAULT_TABLE table=default.Table_252a71de31274362a56b0a5b528f14e9 [id=d89b158670e6490b97c1d0be25662838]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.516793 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_172dcf8ef25b424ca247c507a87c8d0f [id=427b1c6d3389420288bb30b1a4cc8b4f] alter complete (version 1)
I20250115 14:52:15.526639 447 master_service.cc:946] ReplaceTablet: received request to replace tablet b7c6c8d5c58142c0af3e726ea57207b9 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:15.530872 1164 raft_consensus.cc:357] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.531080 1164 raft_consensus.cc:383] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.531132 1164 raft_consensus.cc:738] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.531242 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd745bda7c614d72baab9ec7ccaa3239. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.531297 1164 consensus_queue.cc:260] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.531766 1164 ts_tablet_manager.cc:1428] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.114s user 0.001s sys 0.001s
I20250115 14:52:15.536945 768 raft_consensus.cc:3054] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:15.542737 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6978223c1984a38bec5f2d06a9cf457" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.542894 1164 tablet_bootstrap.cc:492] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:15.544457 984 leader_election.cc:343] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.544593 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5aa92f75f69448d9c43cc808c621f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.544581 984 leader_election.cc:304] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.544847 2563 raft_consensus.cc:2743] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.546677 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c68d7ac544c4606bacabc601f6675f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.494189 879 tablet_service.cc:1467] Processing CreateTablet for tablet f5aa92f75f69448d9c43cc808c621f68 (DEFAULT_TABLE table=default.Table_a0070c4d1be947df84bf19c1efb88fa1 [id=84efaff9f0d44937a7379bf76905e838]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.548507 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeb5ad85b4524994bba041dfb3fc8e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.548527 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5aa92f75f69448d9c43cc808c621f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.550807 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet f06544890cbf460cbe218690e998eb1d (table default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]) successfully deleted
I20250115 14:52:15.496140 888 tablet_service.cc:1467] Processing CreateTablet for tablet 8c91f5a41ea3443e88a8aaae9aa42a18 (DEFAULT_TABLE table=default.Table_0b03e1670f0c48dab3a8f103e6bf6dbd [id=1eeea7ded7c8454cacec6e9802df0a1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.496446 1167 ts_tablet_manager.cc:1905] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:15.552553 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c91f5a41ea3443e88a8aaae9aa42a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.552652 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64d0b02274c47b9ba0ce12dac486031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.499560 885 tablet_service.cc:1467] Processing CreateTablet for tablet 678c0a7256ce4bd19a84de83e2e8633c (DEFAULT_TABLE table=default.Table_5e15a5e365604ac2a0fc0ac6631a167f [id=e9c9601885f148d49cc405a65494f2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.553534 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678c0a7256ce4bd19a84de83e2e8633c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.496842 883 tablet_service.cc:1467] Processing CreateTablet for tablet 554990a441034b25afd6f7d74505e76d (DEFAULT_TABLE table=default.Table_46ab5287629b4006b72bb93ccde2b813 [id=d2548227f1fd4378a61bdc488374821a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.556603 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 554990a441034b25afd6f7d74505e76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.501349 884 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2ffb75f16c4244b795c21f8b89d32a (DEFAULT_TABLE table=default.Table_d384734a041a4ebf963744b6f24a2572 [id=64678b18e5bd42c28d03b195f3603d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.556886 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2ffb75f16c4244b795c21f8b89d32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.558715 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c4c683d7c040bcaa13ff73f9f16613. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.502561 882 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce36c783994c3badf985b79583c0a2 (DEFAULT_TABLE table=default.Table_54c4b2a5cfdc4d65b61e0bf4854a48b8 [id=fe667afae12f49bd93e7ef66aed0c1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.560532 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ce36c783994c3badf985b79583c0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.563877 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb6c5005cfec426f94e401cfee6cf1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.564209 2403 raft_consensus.cc:491] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.503974 881 tablet_service.cc:1467] Processing CreateTablet for tablet eeb5ad85b4524994bba041dfb3fc8e1f (DEFAULT_TABLE table=default.Table_4980e328dcce4600a482147b7a9c0b30 [id=ce371a899b4543f6bdbfe5868d65d714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.564335 2403 raft_consensus.cc:513] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.564523 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeb5ad85b4524994bba041dfb3fc8e1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.564953 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06aa85a23e9e435d9d0d91e807b47a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.505183 878 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb0195969e44d8d9d770c8b0d9c85bf (DEFAULT_TABLE table=default.Table_15773744107143d6b1e5215c832f0a3f [id=f4305d1bfe12463fa68fb1405be5267f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.506606 893 tablet_service.cc:1467] Processing CreateTablet for tablet 063d0e08a8a949f28c0ac62b2e4ec346 (DEFAULT_TABLE table=default.Table_9ff76eab7cfc4904a16a9e1b4ff89486 [id=f4009f6c09084261a84edc3cfedef906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.568565 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb0195969e44d8d9d770c8b0d9c85bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.568766 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063d0e08a8a949f28c0ac62b2e4ec346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.511240 892 tablet_service.cc:1467] Processing CreateTablet for tablet d64d0b02274c47b9ba0ce12dac486031 (DEFAULT_TABLE table=default.Table_e1c85cd6cf714918b357548b0bc26251 [id=233fdd3693334589b22b15dc145108ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.569043 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d64d0b02274c47b9ba0ce12dac486031. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.512645 891 tablet_service.cc:1467] Processing CreateTablet for tablet bd745bda7c614d72baab9ec7ccaa3239 (DEFAULT_TABLE table=default.Table_adabfc70f8eb4826ae67c16382a57ee9 [id=5ae85c348b9c42afb229c07c320291f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.517519 880 tablet_service.cc:1467] Processing CreateTablet for tablet dfa26077522d4641b0e06cb06c6d8dcf (DEFAULT_TABLE table=default.Table_597d1c61c9bc4e398eaf813ad56c8841 [id=411906b36e384ab29c6e4fa709fa44a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.520767 877 tablet_service.cc:1467] Processing CreateTablet for tablet 6c68d7ac544c4606bacabc601f6675f7 (DEFAULT_TABLE table=default.Table_141efb598ef84dfdace28d40f3843d86 [id=6fb7f3061f0041e0a834ad48fdc1f8e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.522739 890 tablet_service.cc:1467] Processing CreateTablet for tablet a3c4c683d7c040bcaa13ff73f9f16613 (DEFAULT_TABLE table=default.Table_84014c06851e40abab7729b00a5aac85 [id=fa8760aa8b62455e942ab02135a67c47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.524039 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ee8b63cc21145d8993b058d8c6d23c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:15.572599 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd745bda7c614d72baab9ec7ccaa3239. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.525243 874 tablet_service.cc:1467] Processing CreateTablet for tablet cb6c5005cfec426f94e401cfee6cf1c1 (DEFAULT_TABLE table=default.Table_252a71de31274362a56b0a5b528f14e9 [id=d89b158670e6490b97c1d0be25662838]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.572778 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c4c683d7c040bcaa13ff73f9f16613. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.572959 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb6c5005cfec426f94e401cfee6cf1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.573189 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c68d7ac544c4606bacabc601f6675f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.576380 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa26077522d4641b0e06cb06c6d8dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.585477 2403 leader_election.cc:290] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.589594 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 1472fd4b71d04d3f9a012f86d7fd0a1a (DEFAULT_TABLE table=default.Table_961a6a3237c54002bab6a3b30ca9301e [id=a750385ac6784d618082f45d5f824c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.590054 1168 tablet_bootstrap.cc:492] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:15.590173 1168 ts_tablet_manager.cc:1397] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.000s
I20250115 14:52:15.590894 1164 tablet_bootstrap.cc:654] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.592541 1168 raft_consensus.cc:357] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.592711 1168 raft_consensus.cc:383] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.592751 1168 raft_consensus.cc:738] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.593019 1168 consensus_queue.cc:260] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.593551 1168 ts_tablet_manager.cc:1428] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:15.593852 1168 tablet_bootstrap.cc:492] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:15.595199 1168 tablet_bootstrap.cc:654] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.597163 2403 raft_consensus.cc:491] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.597285 2403 raft_consensus.cc:513] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.597800 2403 leader_election.cc:290] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.599752 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 5c635d3827ee4dc9a147853e7880a433 (DEFAULT_TABLE table=default.Table_cce291ff531b4d4dbf028cbdb72cc021 [id=7eb1a37936a04a40a504bdddca593150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.600168 1024 tablet_service.cc:1467] Processing CreateTablet for tablet e4e3ad88d9de4077835df6c06db472c8 (DEFAULT_TABLE table=default.Table_9480f0c225c34cef9d44fc404aa3cefd [id=993313229db244f7b7a05166e7ece1c0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:15.595001 710 leader_election.cc:343] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 005a4cd0e9f14e76b01afa49d946ee32
I20250115 14:52:15.602012 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c (DEFAULT_TABLE table=default.Table_c0c0ea95429d42b58100f5fd5f95aabb [id=cc88e0e9080c44f8b53342f1168878bd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:15.602084 710 leader_election.cc:343] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 005a4cd0e9f14e76b01afa49d946ee32
I20250115 14:52:15.602643 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298fdd0217e4df6bfbafc49b2f2c65c" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:15.603365 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1298fdd0217e4df6bfbafc49b2f2c65c" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:15.602186 710 leader_election.cc:304] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:15.607286 2403 raft_consensus.cc:2743] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.611943 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 266521a877ba4fd2bc4914e500f65eae
I20250115 14:52:15.613391 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a3b0bb374dc54533b7e8a5e5369c2f39
W20250115 14:52:15.613618 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e2ffb75f16c4244b795c21f8b89d32a consensus state has no leader
I20250115 14:52:15.613979 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0d800cc4760a43fc82ef693a68ef4928" } modify_external_catalogs: true
I20250115 14:52:15.614189 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0d800cc4760a43fc82ef693a68ef4928" } modify_external_catalogs: true
I20250115 14:52:15.614114 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_186b04ca0bf6418895c667086df34188"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.614655 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_186b04ca0bf6418895c667086df34188 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.615383 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_669581139cfa4cffa25ef15e5285c455" } new_table_name: "default.Table_fab8bb6cb2c0417d9a6e6f8a9f09ab16" modify_external_catalogs: true
I20250115 14:52:15.615551 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4a036dcd4a7a41fab5a082306bba837b" } new_table_name: "default.Table_d35dcbcdfd8646c681514758fd32be46" modify_external_catalogs: true
I20250115 14:52:15.616703 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2923dd60927d49f5b0ca09438cfd4ce3
I20250115 14:52:15.616797 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a5018e358d234ea28023da76a8c61a86
I20250115 14:52:15.617295 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 266521a877ba4fd2bc4914e500f65eae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.617293 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4f994af897944da0b4e1f9cf4a90e137" } new_table_name: "default.Table_9075831c659548d089b509d540c97b07" modify_external_catalogs: true
I20250115 14:52:15.617376 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c3dbb94e26f1439ba0bb576d883ec848"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:15.617477 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b0bb374dc54533b7e8a5e5369c2f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:15.617777 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3dbb94e26f1439ba0bb576d883ec848 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.618502 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 2923dd60927d49f5b0ca09438cfd4ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.618693 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet a5018e358d234ea28023da76a8c61a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.621088 1168 tablet_bootstrap.cc:492] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:15.621280 1168 ts_tablet_manager.cc:1397] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20250115 14:52:15.622030 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c62ee465977947a19aba3314a3e2a3f7" } new_table_name: "default.Table_c7251196621d49aa966653f5a456c7eb" modify_external_catalogs: true
I20250115 14:52:15.626050 756 tablet_service.cc:1514] Processing DeleteTablet for tablet 266521a877ba4fd2bc4914e500f65eae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.626248 755 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b0bb374dc54533b7e8a5e5369c2f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.627271 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5c635d3827ee4dc9a147853e7880a433 (DEFAULT_TABLE table=default.Table_cce291ff531b4d4dbf028cbdb72cc021 [id=7eb1a37936a04a40a504bdddca593150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.627568 742 tablet_service.cc:1467] Processing CreateTablet for tablet 1472fd4b71d04d3f9a012f86d7fd0a1a (DEFAULT_TABLE table=default.Table_961a6a3237c54002bab6a3b30ca9301e [id=a750385ac6784d618082f45d5f824c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.627669 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c635d3827ee4dc9a147853e7880a433. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.627910 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1472fd4b71d04d3f9a012f86d7fd0a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.628799 743 tablet_service.cc:1467] Processing CreateTablet for tablet e4e3ad88d9de4077835df6c06db472c8 (DEFAULT_TABLE table=default.Table_9480f0c225c34cef9d44fc404aa3cefd [id=993313229db244f7b7a05166e7ece1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.629592 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4e3ad88d9de4077835df6c06db472c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.629632 737 tablet_service.cc:1467] Processing CreateTablet for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c (DEFAULT_TABLE table=default.Table_c0c0ea95429d42b58100f5fd5f95aabb [id=cc88e0e9080c44f8b53342f1168878bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.630577 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.621964 1168 raft_consensus.cc:357] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.633051 1168 raft_consensus.cc:383] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.636893 773 raft_consensus.cc:3054] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:15.636938 1166 ts_tablet_manager.cc:1918] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:15.637233 1166 log.cc:1198] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/4ee8b63cc21145d8993b058d8c6d23c7
I20250115 14:52:15.637634 1166 ts_tablet_manager.cc:1939] T 4ee8b63cc21145d8993b058d8c6d23c7 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:15.639734 710 leader_election.cc:304] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:15.640039 2403 raft_consensus.cc:2743] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.640380 741 tablet_service.cc:1514] Processing DeleteTablet for tablet 2923dd60927d49f5b0ca09438cfd4ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.641801 747 tablet_service.cc:1514] Processing DeleteTablet for tablet a5018e358d234ea28023da76a8c61a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.645812 1168 raft_consensus.cc:738] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.646096 1168 consensus_queue.cc:260] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.646875 1168 ts_tablet_manager.cc:1428] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.025s user 0.001s sys 0.000s
I20250115 14:52:15.647172 1168 tablet_bootstrap.cc:492] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:15.648828 1168 tablet_bootstrap.cc:654] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.655300 774 raft_consensus.cc:3054] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:15.656445 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 964a7d57633841dcb04ed07a25de4ad1 (DEFAULT_TABLE table=default.Table_bdb6b9f92fac4cb49ce119752a15b45e [id=0c33cd51b93a430ba33b24df0a0f6719]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:15.656842 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet f2993e1c95234a1db2ae98a8e08c43f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2993e1c95234a1db2ae98a8e08c43f8 already in progress: creating tablet
I20250115 14:52:15.658519 1167 ts_tablet_manager.cc:1918] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:15.662721 1164 tablet_bootstrap.cc:492] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.663784 1164 ts_tablet_manager.cc:1397] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.121s user 0.005s sys 0.005s
I20250115 14:52:15.664322 2763 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:15.664801 2563 raft_consensus.cc:491] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.664942 2563 raft_consensus.cc:513] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.665539 2563 leader_election.cc:290] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.663661 1167 log.cc:1198] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/5265d5eeac83481ca97801e3475886da
I20250115 14:52:15.668754 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f19f0a73ca24a37a5ef8e19b86ccc3c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:15.669255 983 leader_election.cc:343] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.670084 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f19f0a73ca24a37a5ef8e19b86ccc3c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.670166 1167 ts_tablet_manager.cc:1939] T 5265d5eeac83481ca97801e3475886da P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
W20250115 14:52:15.670855 984 leader_election.cc:343] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.670953 984 leader_election.cc:304] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.671289 2755 raft_consensus.cc:2743] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.671959 2763 tablet.cc:1721] T 8c7e70288eb248f1a71646c66e376a63 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:15.655658 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 4ee8b63cc21145d8993b058d8c6d23c7 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]) successfully deleted
I20250115 14:52:15.666064 1164 raft_consensus.cc:357] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
W20250115 14:52:15.675218 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 266521a877ba4fd2bc4914e500f65eae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 266521a877ba4fd2bc4914e500f65eae already in progress: creating tablet
I20250115 14:52:15.675447 1164 raft_consensus.cc:383] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:52:15.675730 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a3b0bb374dc54533b7e8a5e5369c2f39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3b0bb374dc54533b7e8a5e5369c2f39 already in progress: creating tablet
I20250115 14:52:15.675921 1164 raft_consensus.cc:738] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.676249 2549 raft_consensus.cc:491] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:52:15.676218 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2923dd60927d49f5b0ca09438cfd4ce3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2923dd60927d49f5b0ca09438cfd4ce3 already in progress: creating tablet
I20250115 14:52:15.676429 2549 raft_consensus.cc:513] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.676983 2549 leader_election.cc:290] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:15.677673 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36de948083a946eba1a96f276e4f4f91" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:15.677850 1042 raft_consensus.cc:2463] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:15.678277 847 leader_election.cc:304] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:15.678808 2549 raft_consensus.cc:2798] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:15.678879 2549 raft_consensus.cc:491] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:15.678930 2549 raft_consensus.cc:3054] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:15.679870 2657 raft_consensus.cc:491] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.679972 2657 raft_consensus.cc:513] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.680472 2657 leader_election.cc:290] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:15.681588 2549 raft_consensus.cc:513] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:15.681763 848 leader_election.cc:343] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:15.682492 847 leader_election.cc:343] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 9f4475441e4a467a8d593b742d2a5b80
W20250115 14:52:15.690384 2782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250115 14:52:15.693130 2631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250115 14:52:15.696767 1164 consensus_queue.cc:260] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.701122 2631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250115 14:52:15.704491 2631 hybrid_clock.cc:648] HybridClock initialized: now 1736952735704336 us; error 116 us; skew 500 ppm
I20250115 14:52:15.705010 2631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250115 14:52:15.706611 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a5018e358d234ea28023da76a8c61a86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5018e358d234ea28023da76a8c61a86 already in progress: creating tablet
I20250115 14:52:15.709757 742 tablet_service.cc:1467] Processing CreateTablet for tablet 964a7d57633841dcb04ed07a25de4ad1 (DEFAULT_TABLE table=default.Table_bdb6b9f92fac4cb49ce119752a15b45e [id=0c33cd51b93a430ba33b24df0a0f6719]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.710187 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 964a7d57633841dcb04ed07a25de4ad1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:15.714587 848 consensus_peers.cc:487] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:15.717761 2755 raft_consensus.cc:491] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.717907 2755 raft_consensus.cc:513] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.714514 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1210) took 1621 ms. Trace:
I20250115 14:52:15.718513 2755 leader_election.cc:290] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.719147 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c04fa23e7c40a5b5e07852e5d74a27" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:15.720348 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 7dffe3941dcd409cbbd64efefe5831ba (DEFAULT_TABLE table=default.Table_e25f23f07b65483aa5d163d8017a19f1 [id=887a9cbe87684db5a474efcdbf3c2d6c]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:15.720479 983 leader_election.cc:343] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.719653 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c04fa23e7c40a5b5e07852e5d74a27" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:15.721150 984 leader_election.cc:343] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.721256 984 leader_election.cc:304] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.721581 2563 raft_consensus.cc:491] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.719383 1164 ts_tablet_manager.cc:1428] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20250115 14:52:15.721678 2563 raft_consensus.cc:513] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.722204 2563 leader_election.cc:290] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.722332 2563 raft_consensus.cc:2743] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.723400 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d564ec5da4e4a3a9b903a4d9bbdb63a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:15.723937 983 leader_election.cc:343] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.724553 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d564ec5da4e4a3a9b903a4d9bbdb63a" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:15.725059 984 leader_election.cc:343] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.725159 984 leader_election.cc:304] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.726078 2755 raft_consensus.cc:2743] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.729048 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 4adad3b609e8419890d463a66edc2189 (DEFAULT_TABLE table=default.Table_5392236e7689487c8c5909b627a06a12 [id=1f8a0d356ddd486e8faf98b98854afeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.718505 1167 rpcz_store.cc:276] 0115 14:52:14.092531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.092578 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:15.462348 (+1369770us) ts_tablet_manager.cc:1145] Deleting tablet 5265d5eeac83481ca97801e3475886da
0115 14:52:15.462357 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.555263 (+ 92906us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:15.655335 (+100072us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:15.713722 (+ 58387us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369578}
I20250115 14:52:15.731629 2549 leader_election.cc:290] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:15.732620 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36de948083a946eba1a96f276e4f4f91" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:15.732908 1046 raft_consensus.cc:3054] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:15.733129 1168 tablet_bootstrap.cc:492] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:15.733842 1168 ts_tablet_manager.cc:1397] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.003s
W20250115 14:52:15.727725 848 leader_election.cc:343] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.736862 2631 webserver.cc:458] Webserver started at http://127.0.104.60:37445/ using document root <none> and password file <none>
I20250115 14:52:15.737917 2631 fs_manager.cc:362] Metadata directory not provided
I20250115 14:52:15.738059 2631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250115 14:52:15.738802 1164 tablet_bootstrap.cc:492] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:15.737157 848 leader_election.cc:304] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.743491 1010 tablet_service.cc:1467] Processing CreateTablet for tablet ca04254f3b8846d490e5277f3d995069 (DEFAULT_TABLE table=default.Table_58d9a1abb0be4eaf8ebc5c55369a0154 [id=f28c9084fd1247d3be5dfef058d79a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.744441 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7dffe3941dcd409cbbd64efefe5831ba (DEFAULT_TABLE table=default.Table_e25f23f07b65483aa5d163d8017a19f1 [id=887a9cbe87684db5a474efcdbf3c2d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.744846 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dffe3941dcd409cbbd64efefe5831ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.744916 2549 raft_consensus.cc:2743] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.745622 1164 tablet_bootstrap.cc:654] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.746522 2631 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.002s
I20250115 14:52:15.745290 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1211) took 1476 ms. Trace:
I20250115 14:52:15.750034 743 tablet_service.cc:1467] Processing CreateTablet for tablet 4adad3b609e8419890d463a66edc2189 (DEFAULT_TABLE table=default.Table_5392236e7689487c8c5909b627a06a12 [id=1f8a0d356ddd486e8faf98b98854afeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.750388 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4adad3b609e8419890d463a66edc2189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.752563 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b7c6c8d5c58142c0af3e726ea57207b9
I20250115 14:52:15.753002 447 catalog_manager.cc:6596] ReplaceTablet: tablet b7c6c8d5c58142c0af3e726ea57207b9 deleted and replaced by tablet 8c33907919374a6fbf9e7175c7c9a485
I20250115 14:52:15.754086 737 tablet_service.cc:1467] Processing CreateTablet for tablet ca04254f3b8846d490e5277f3d995069 (DEFAULT_TABLE table=default.Table_58d9a1abb0be4eaf8ebc5c55369a0154 [id=f28c9084fd1247d3be5dfef058d79a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.754426 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca04254f3b8846d490e5277f3d995069. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.748430 1167 rpcz_store.cc:276] 0115 14:52:14.268153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.270899 (+ 2746us) service_pool.cc:225] Handling call
0115 14:52:15.738023 (+1467124us) ts_tablet_manager.cc:1145] Deleting tablet bfc6e5c81a6d40a3a2054450d0478ad6
0115 14:52:15.738881 (+ 858us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.744511 (+ 5630us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465937}
I20250115 14:52:15.757866 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c6c8d5c58142c0af3e726ea57207b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c33907919374a6fbf9e7175c7c9a485) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:15.759727 1168 raft_consensus.cc:357] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.759941 1168 raft_consensus.cc:383] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.759999 1168 raft_consensus.cc:738] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.760201 1168 consensus_queue.cc:260] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.760751 1168 ts_tablet_manager.cc:1428] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20250115 14:52:15.761035 1168 tablet_bootstrap.cc:492] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:15.762406 1164 tablet_bootstrap.cc:492] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.762583 1164 ts_tablet_manager.cc:1397] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.005s
I20250115 14:52:15.763319 1164 raft_consensus.cc:357] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.763557 1164 raft_consensus.cc:383] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.763649 1164 raft_consensus.cc:738] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.763927 1164 consensus_queue.cc:260] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.764858 1164 ts_tablet_manager.cc:1428] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:15.765708 1164 tablet_bootstrap.cc:492] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:15.766068 1168 tablet_bootstrap.cc:654] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.767355 1164 tablet_bootstrap.cc:654] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.767731 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1218) took 1273 ms. Trace:
W20250115 14:52:15.767585 848 consensus_peers.cc:487] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:15.736802 1046 raft_consensus.cc:2463] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:15.777072 1168 tablet_bootstrap.cc:492] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:15.777211 1168 ts_tablet_manager.cc:1397] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20250115 14:52:15.777886 1168 raft_consensus.cc:357] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.770748 1167 rpcz_store.cc:276] 0115 14:52:14.493850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.494173 (+ 323us) service_pool.cc:225] Handling call
0115 14:52:15.762393 (+1268220us) ts_tablet_manager.cc:1145] Deleting tablet 061234cb76b842a7a374ddd5f44fd9a1
0115 14:52:15.762880 (+ 487us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.766962 (+ 4082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":931602}
I20250115 14:52:15.778072 1168 raft_consensus.cc:383] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.778489 1168 raft_consensus.cc:738] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.778762 1168 consensus_queue.cc:260] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.779471 847 leader_election.cc:304] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:15.779901 2748 raft_consensus.cc:2798] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:15.780005 2748 raft_consensus.cc:695] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:15.780226 2748 consensus_queue.cc:237] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:15.783651 984 consensus_peers.cc:487] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:15.783825 2549 consensus_queue.cc:1035] T 5df0250d08214a9783c392071126ed4a P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:15.786804 737 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c6c8d5c58142c0af3e726ea57207b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c33907919374a6fbf9e7175c7c9a485) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.787143 1168 ts_tablet_manager.cc:1428] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250115 14:52:15.787266 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1220) took 1285 ms. Trace:
I20250115 14:52:15.790024 1025 tablet_service.cc:1467] Processing CreateTablet for tablet edff9d5a740d4d5d8df02f4549c30700 (DEFAULT_TABLE table=default.Table_7bb0ca02228940ee91c77404cb564a29 [id=8f9b630f463949249cbb6088dfdc2bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.787580 1168 tablet_bootstrap.cc:492] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:15.792078 1168 tablet_bootstrap.cc:654] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.796280 2810 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250115 14:52:15.797433 1164 tablet_bootstrap.cc:492] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.797577 1164 ts_tablet_manager.cc:1397] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.011s
I20250115 14:52:15.798270 1164 raft_consensus.cc:357] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.798460 1164 raft_consensus.cc:383] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.798520 1164 raft_consensus.cc:738] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.798748 1164 consensus_queue.cc:260] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.799258 1164 ts_tablet_manager.cc:1428] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:15.800494 2631 fs_manager.cc:730] Time spent opening block manager: real 0.052s user 0.004s sys 0.004s
I20250115 14:52:15.800701 2631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data,/tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
uuid: "1cc2e23c560f4096be7b14006bdfc525"
format_stamp: "Formatted at 2025-01-15 14:51:48 on dist-test-slave-pxr4"
I20250115 14:52:15.790714 1167 rpcz_store.cc:276] 0115 14:52:14.501661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.501716 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:15.782418 (+1280702us) ts_tablet_manager.cc:1145] Deleting tablet 9ad3cda1056349a789a1751c5d386d0c
0115 14:52:15.782811 (+ 393us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.786496 (+ 3685us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":951016}
I20250115 14:52:15.803149 2631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250115 14:52:15.808135 1164 tablet_bootstrap.cc:492] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:15.813083 1019 tablet_service.cc:1467] Processing CreateTablet for tablet 77863c5591e64a38926dcbd64c46437d (DEFAULT_TABLE table=default.Table_9be6523788f347129832e1de3230c4c7 [id=3cc080a5cca04dc285ce4dd4122faa8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.815013 1164 tablet_bootstrap.cc:654] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.812932 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1221) took 1310 ms. Trace:
I20250115 14:52:15.823805 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6bb430490bfb46a18988ae5c2f2707a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:15.824051 737 tablet_service.cc:1467] Processing CreateTablet for tablet edff9d5a740d4d5d8df02f4549c30700 (DEFAULT_TABLE table=default.Table_7bb0ca02228940ee91c77404cb564a29 [id=8f9b630f463949249cbb6088dfdc2bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.824345 743 tablet_service.cc:1467] Processing CreateTablet for tablet 77863c5591e64a38926dcbd64c46437d (DEFAULT_TABLE table=default.Table_9be6523788f347129832e1de3230c4c7 [id=3cc080a5cca04dc285ce4dd4122faa8d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:15.825361 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bb430490bfb46a18988ae5c2f2707a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.816671 1167 rpcz_store.cc:276] 0115 14:52:14.501790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.501836 (+ 46us) service_pool.cc:225] Handling call
0115 14:52:15.806602 (+1304766us) ts_tablet_manager.cc:1145] Deleting tablet 13a4837b72eb4008ad612a9fc86dfac7
0115 14:52:15.807096 (+ 494us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.812129 (+ 5033us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":974792}
I20250115 14:52:15.825502 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edff9d5a740d4d5d8df02f4549c30700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.825794 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77863c5591e64a38926dcbd64c46437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.832567 1164 tablet_bootstrap.cc:492] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.832760 1164 ts_tablet_manager.cc:1397] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.002s
I20250115 14:52:15.833549 1164 raft_consensus.cc:357] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.834012 1164 raft_consensus.cc:383] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.834125 1164 raft_consensus.cc:738] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.834407 1164 consensus_queue.cc:260] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.835232 1164 ts_tablet_manager.cc:1428] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:15.835530 1164 tablet_bootstrap.cc:492] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:15.837846 1164 tablet_bootstrap.cc:654] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.839644 2549 consensus_queue.cc:1035] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:15.837066 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1222) took 1334 ms. Trace:
I20250115 14:52:15.843524 1168 tablet_bootstrap.cc:492] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:15.843653 1168 ts_tablet_manager.cc:1397] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.001s
I20250115 14:52:15.844417 1168 raft_consensus.cc:357] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.844590 1168 raft_consensus.cc:383] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.844642 1168 raft_consensus.cc:738] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.845436 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36de948083a946eba1a96f276e4f4f91" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.846951 742 tablet_service.cc:1467] Processing CreateTablet for tablet 082920cb7a7f4994af352d59678518a4 (DEFAULT_TABLE table=default.Table_095871dcaa2547f8a7d859237e9bf4fa [id=5be54673655e4263b5f3fd9fe48b06a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.847405 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 082920cb7a7f4994af352d59678518a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.848565 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f4475441e4a467a8d593b742d2a5b80" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.840745 1167 rpcz_store.cc:276] 0115 14:52:14.501894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.502151 (+ 257us) service_pool.cc:225] Handling call
0115 14:52:15.830006 (+1327855us) ts_tablet_manager.cc:1145] Deleting tablet 24729156769443f8926577bae7707847
0115 14:52:15.830872 (+ 866us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.836159 (+ 5287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":998099}
I20250115 14:52:15.849242 743 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6eacb0c16f4c4ca8db8eb8f65be1ae (DEFAULT_TABLE table=default.Table_cf227e000a9a4a8f8565e985bb801c61 [id=198fa58e5e3f4e87bcf297e5f1c69fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.849586 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6eacb0c16f4c4ca8db8eb8f65be1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.851981 1164 tablet_bootstrap.cc:492] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.852098 1164 ts_tablet_manager.cc:1397] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20250115 14:52:15.852936 1164 raft_consensus.cc:357] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.853106 1164 raft_consensus.cc:383] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.853154 1164 raft_consensus.cc:738] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.853363 1164 consensus_queue.cc:260] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:15.854543 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36de948083a946eba1a96f276e4f4f91" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:15.854290 1164 ts_tablet_manager.cc:1428] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250115 14:52:15.854781 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b7c6c8d5c58142c0af3e726ea57207b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7c6c8d5c58142c0af3e726ea57207b9 already in progress: creating tablet
I20250115 14:52:15.855531 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 5265d5eeac83481ca97801e3475886da (table default.Table_4add05103df84280812bf61910d76d1d [id=47db9a9013ec4a2cb3cc0cc8f9caba12]) successfully deleted
I20250115 14:52:15.856053 1164 tablet_bootstrap.cc:492] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:15.857646 1164 tablet_bootstrap.cc:654] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:15.861431 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet bfc6e5c81a6d40a3a2054450d0478ad6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfc6e5c81a6d40a3a2054450d0478ad6 already in progress: creating tablet
I20250115 14:52:15.858517 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1217) took 1363 ms. Trace:
W20250115 14:52:15.864010 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1218f0bdb1e14bd78d76c6a06805f38c consensus state has no leader
I20250115 14:52:15.863898 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1401e274205941c5bd2a45b1edf58461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:15.864409 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5beff98980534f9694b89e4fa022ab17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.864904 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5beff98980534f9694b89e4fa022ab17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:15.865109 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1401e274205941c5bd2a45b1edf58461 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.865190 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b (table default.Table_f634603ea42b4215b8ef3df8c446ada7 [id=174b82e1e06f4073a74a25aad8fc8045]). Expected version 2 got 0
I20250115 14:52:15.865383 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 5df0250d08214a9783c392071126ed4a (table default.Table_90c834d5ef634e6aa797dd220e3d5faa [id=8ffbe34c2a11417f8ca50dbdd627bf22]). Expected version 2 got 0
W20250115 14:52:15.864429 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 061234cb76b842a7a374ddd5f44fd9a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 061234cb76b842a7a374ddd5f44fd9a1 already in progress: creating tablet
I20250115 14:52:15.867532 450 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
I20250115 14:52:15.868810 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_36b868736e9e46528bddc87ba5998ff8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:15.869239 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36b868736e9e46528bddc87ba5998ff8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.870360 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00
I20250115 14:52:15.862723 1167 rpcz_store.cc:276] 0115 14:52:14.493739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.494050 (+ 311us) service_pool.cc:225] Handling call
0115 14:52:15.852532 (+1358482us) ts_tablet_manager.cc:1145] Deleting tablet 7830c683a109448ab06c15b5de692e8b
0115 14:52:15.853383 (+ 851us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.857639 (+ 4256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017354}
I20250115 14:52:15.860769 1168 consensus_queue.cc:260] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.879055 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1243) took 1211 ms. Trace:
I20250115 14:52:15.885156 883 tablet_service.cc:1467] Processing CreateTablet for tablet 1472fd4b71d04d3f9a012f86d7fd0a1a (DEFAULT_TABLE table=default.Table_961a6a3237c54002bab6a3b30ca9301e [id=a750385ac6784d618082f45d5f824c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.885561 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1472fd4b71d04d3f9a012f86d7fd0a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.904606 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1298fdd0217e4df6bfbafc49b2f2c65c with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0195969e44d8d9d770c8b0d9c85bf) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.904861 1166 tablet_replica.cc:331] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:15.904981 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 005a4cd0e9f14e76b01afa49d946ee32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.905053 1166 raft_consensus.cc:2238] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:15.905246 1166 raft_consensus.cc:2267] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:15.907792 1166 ts_tablet_manager.cc:1905] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:15.908624 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ebfbdc698c8c448ca2e60757b1fd574c" } modify_external_catalogs: true
I20250115 14:52:15.908820 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ebfbdc698c8c448ca2e60757b1fd574c" } modify_external_catalogs: true
I20250115 14:52:15.910099 742 tablet_service.cc:1467] Processing CreateTablet for tablet 2c519f995c4c413c85be48c4577d7db2 (DEFAULT_TABLE table=default.Table_adcc45a64d8840fba4e22623a802abb6 [id=aded3d5cc1c64969bc8528c9b5e038af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.910517 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c519f995c4c413c85be48c4577d7db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.881955 1167 rpcz_store.cc:276] 0115 14:52:14.667067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.869631 (+202564us) service_pool.cc:225] Handling call
0115 14:52:15.874025 (+1004394us) ts_tablet_manager.cc:1145] Deleting tablet 3e834623f9c7482d965afd23c89a2510
0115 14:52:15.874356 (+ 331us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.878423 (+ 4067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1003297}
I20250115 14:52:15.915218 1166 ts_tablet_manager.cc:1918] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:15.915318 1166 log.cc:1198] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/1298fdd0217e4df6bfbafc49b2f2c65c
I20250115 14:52:15.915632 1166 ts_tablet_manager.cc:1939] T 1298fdd0217e4df6bfbafc49b2f2c65c P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:15.915968 877 tablet_service.cc:1467] Processing CreateTablet for tablet 5c635d3827ee4dc9a147853e7880a433 (DEFAULT_TABLE table=default.Table_cce291ff531b4d4dbf028cbdb72cc021 [id=7eb1a37936a04a40a504bdddca593150]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.916412 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c635d3827ee4dc9a147853e7880a433. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.916481 1166 tablet_replica.cc:331] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:15.916640 1166 raft_consensus.cc:2238] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:15.916787 1166 raft_consensus.cc:2267] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:15.879617 1168 ts_tablet_manager.cc:1428] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.036s user 0.003s sys 0.000s
I20250115 14:52:15.917273 1166 ts_tablet_manager.cc:1905] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:15.917518 1168 tablet_bootstrap.cc:492] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:15.919268 1168 tablet_bootstrap.cc:654] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.922734 1166 ts_tablet_manager.cc:1918] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:15.922852 1166 log.cc:1198] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/005a4cd0e9f14e76b01afa49d946ee32
I20250115 14:52:15.923105 1166 ts_tablet_manager.cc:1939] T 005a4cd0e9f14e76b01afa49d946ee32 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:15.932585 2563 consensus_queue.cc:1035] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:15.933748 893 tablet_service.cc:1467] Processing CreateTablet for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c (DEFAULT_TABLE table=default.Table_c0c0ea95429d42b58100f5fd5f95aabb [id=cc88e0e9080c44f8b53342f1168878bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.934142 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.941473 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 266521a877ba4fd2bc4914e500f65eae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:15.942489 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 1298fdd0217e4df6bfbafc49b2f2c65c (table default.Table_15773744107143d6b1e5215c832f0a3f [id=f4305d1bfe12463fa68fb1405be5267f]) successfully deleted
I20250115 14:52:15.943082 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 005a4cd0e9f14e76b01afa49d946ee32 (table default.Table_d611f6594f554cfe99c5aa5f2f7ea0e2 [id=cfab75a29f6c4e388b90e1814fa72fae]) successfully deleted
I20250115 14:52:15.943989 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:15.944360 1166 tablet_replica.cc:331] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:15.944554 1166 raft_consensus.cc:2238] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:15.944726 1166 raft_consensus.cc:2267] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:15.945302 1166 ts_tablet_manager.cc:1905] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:15.946406 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd745bda7c614d72baab9ec7ccaa3239. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.948458 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 063d0e08a8a949f28c0ac62b2e4ec346. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.949892 748 tablet_service.cc:1467] Processing CreateTablet for tablet bf86a1cd1d7b40ee8c3638af06770b79 (DEFAULT_TABLE table=default.Table_7423aeb75aaf4a9f8d98517868158323 [id=3f1e905acb9148d39fb7b36467184a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.950237 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf86a1cd1d7b40ee8c3638af06770b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.950968 1162 tablet_bootstrap.cc:654] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.951009 743 tablet_service.cc:1467] Processing CreateTablet for tablet 2fcff7017c77487d9ed5fd16f360f4f9 (DEFAULT_TABLE table=default.Table_f7d31cfc0fe54ffdb1b9fc0b8b26fe3e [id=dbeb5e86f26349709ba1761e7eb7d8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.951300 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcff7017c77487d9ed5fd16f360f4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.951622 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c4c683d7c040bcaa13ff73f9f16613. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.952548 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2ffb75f16c4244b795c21f8b89d32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.954407 737 tablet_service.cc:1467] Processing CreateTablet for tablet adcb42acbc7142498155fdb8fd04fb8b (DEFAULT_TABLE table=default.Table_883692e629d34120b90cdbf0744cdfc8 [id=e95f9b6c227c4903b523486a033650e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.954696 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adcb42acbc7142498155fdb8fd04fb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.955178 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1472fd4b71d04d3f9a012f86d7fd0a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.955227 742 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbdf9c664434c4ba24deef67299c018 (DEFAULT_TABLE table=default.Table_67c82d5293bd4157bc5887a9d6410293 [id=efa1ad976e3b4a22ba8866ebf80b6f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.955495 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bbdf9c664434c4ba24deef67299c018. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.960186 750 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea876c00b5f441ba62ced9bb67f3818 (DEFAULT_TABLE table=default.Table_52240b9928824856bd6ce7d2ea06d625 [id=41b963c2c7e14f0fb996bd1de6251e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.960487 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea876c00b5f441ba62ced9bb67f3818. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.961131 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c635d3827ee4dc9a147853e7880a433. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.964475 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4e3ad88d9de4077835df6c06db472c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.965598 2403 raft_consensus.cc:491] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.965723 2403 raft_consensus.cc:513] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.966135 1751 ts_tablet_manager.cc:1918] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:15.966230 1751 log.cc:1198] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/18f23815196047628d562da7312b010f
I20250115 14:52:15.966327 2403 leader_election.cc:290] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:15.966475 1751 ts_tablet_manager.cc:1939] T 18f23815196047628d562da7312b010f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:15.966907 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70425c25919b4d0b999450210e1cb33f" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:15.967170 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1193) took 2429 ms. Trace:
I20250115 14:52:15.967239 1751 rpcz_store.cc:276] 0115 14:52:13.537850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.537906 (+ 56us) service_pool.cc:225] Handling call
0115 14:52:15.358453 (+1820547us) ts_tablet_manager.cc:1145] Deleting tablet 18f23815196047628d562da7312b010f
0115 14:52:15.358462 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:15.958888 (+600426us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:15.966065 (+ 7177us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:15.967153 (+ 1088us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1806578}
I20250115 14:52:15.967619 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 18f23815196047628d562da7312b010f (table default.Table_b053773cca094e6499791ad922f1fe10 [id=bd973eb84bcb426fa3e5963b7cb949dd]) successfully deleted
I20250115 14:52:15.967857 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70425c25919b4d0b999450210e1cb33f" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:15.968408 710 leader_election.cc:304] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:15.968484 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.968627 2403 raft_consensus.cc:2743] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.970551 1166 ts_tablet_manager.cc:1918] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250115 14:52:15.970690 1166 log.cc:1198] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/d13b4c0edc2c4fc9a1c527e1ff6a3d00
I20250115 14:52:15.971093 1166 ts_tablet_manager.cc:1939] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:15.972494 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 964a7d57633841dcb04ed07a25de4ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.973793 1164 tablet_bootstrap.cc:492] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:15.973913 1164 ts_tablet_manager.cc:1397] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.118s user 0.015s sys 0.013s
I20250115 14:52:15.974609 1164 raft_consensus.cc:357] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.974768 1164 raft_consensus.cc:383] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.974820 1164 raft_consensus.cc:738] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.974999 1164 consensus_queue.cc:260] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.976166 1164 ts_tablet_manager.cc:1428] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:15.976473 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dffe3941dcd409cbbd64efefe5831ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.977564 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4adad3b609e8419890d463a66edc2189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.979539 2563 raft_consensus.cc:491] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:15.979646 2563 raft_consensus.cc:513] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.980132 1162 tablet_bootstrap.cc:492] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:15.980679 1164 tablet_bootstrap.cc:492] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:15.981338 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbc2ae77d974b80bc1c93a3e44f3dcd" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:15.982951 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dbc2ae77d974b80bc1c93a3e44f3dcd" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:15.983378 1164 tablet_bootstrap.cc:654] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.980227 2563 leader_election.cc:290] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:15.986086 1162 ts_tablet_manager.cc:1397] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.549s user 0.009s sys 0.015s
I20250115 14:52:15.986955 1162 raft_consensus.cc:357] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.987347 1162 raft_consensus.cc:383] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:15.987409 1162 raft_consensus.cc:738] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:15.987596 1162 consensus_queue.cc:260] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:15.981937 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca04254f3b8846d490e5277f3d995069. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.988353 1162 ts_tablet_manager.cc:1428] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:15.988516 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edff9d5a740d4d5d8df02f4549c30700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.988616 1162 tablet_bootstrap.cc:492] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:15.982151 983 leader_election.cc:343] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:15.988866 984 leader_election.cc:343] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:15.989228 984 leader_election.cc:304] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:15.990170 1162 tablet_bootstrap.cc:654] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:15.994033 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77863c5591e64a38926dcbd64c46437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.994184 2563 raft_consensus.cc:2743] T 1dbc2ae77d974b80bc1c93a3e44f3dcd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:15.996052 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa26077522d4641b0e06cb06c6d8dcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.996627 1025 tablet_service.cc:1467] Processing CreateTablet for tablet 082920cb7a7f4994af352d59678518a4 (DEFAULT_TABLE table=default.Table_095871dcaa2547f8a7d859237e9bf4fa [id=5be54673655e4263b5f3fd9fe48b06a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.996927 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 082920cb7a7f4994af352d59678518a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.997617 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ce36c783994c3badf985b79583c0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:15.998430 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:15.998643 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:45653, user_credentials={real_user=slave}} blocked reactor thread for 52690.5us
I20250115 14:52:15.999624 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_0d800cc4760a43fc82ef693a68ef4928 [id=be99ec07535243a68c1b3ee1be0c62e7]) successfully deleted
W20250115 14:52:16.001703 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f430135d6eb4484af60ee4425c3981c consensus state has no leader
W20250115 14:52:16.004511 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04daad461f0b433ba635df87439b5bc1 consensus state has no leader
I20250115 14:52:15.985703 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6eacb0c16f4c4ca8db8eb8f65be1ae (DEFAULT_TABLE table=default.Table_cf227e000a9a4a8f8565e985bb801c61 [id=198fa58e5e3f4e87bcf297e5f1c69fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:15.998602 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c91f5a41ea3443e88a8aaae9aa42a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.007200 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_0cec3b5c1b834606bcb399747273ec4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.007692 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cec3b5c1b834606bcb399747273ec4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:15.999094 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 2fcff7017c77487d9ed5fd16f360f4f9 (DEFAULT_TABLE table=default.Table_f7d31cfc0fe54ffdb1b9fc0b8b26fe3e [id=dbeb5e86f26349709ba1761e7eb7d8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.006511 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 2c519f995c4c413c85be48c4577d7db2 (DEFAULT_TABLE table=default.Table_adcc45a64d8840fba4e22623a802abb6 [id=aded3d5cc1c64969bc8528c9b5e038af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.007299 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbdf9c664434c4ba24deef67299c018 (DEFAULT_TABLE table=default.Table_67c82d5293bd4157bc5887a9d6410293 [id=efa1ad976e3b4a22ba8866ebf80b6f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.007390 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5130494a392144e7befd2183a015736b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.009100 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3ac29fd7d7614bd98b6ab37861ad016e" } new_table_name: "default.Table_3bf56b835da847af9d4651f729286758" modify_external_catalogs: true
I20250115 14:52:16.009541 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb6c5005cfec426f94e401cfee6cf1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.007905 1014 tablet_service.cc:1467] Processing CreateTablet for tablet adcb42acbc7142498155fdb8fd04fb8b (DEFAULT_TABLE table=default.Table_883692e629d34120b90cdbf0744cdfc8 [id=e95f9b6c227c4903b523486a033650e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.010643 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adcb42acbc7142498155fdb8fd04fb8b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.011085 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb2a6e23cafa4ce99514e063c8084cb4 consensus state has no leader
I20250115 14:52:16.011437 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_def81b30f40f4877a7e329da62ee3b27" } modify_external_catalogs: true
I20250115 14:52:16.011595 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_def81b30f40f4877a7e329da62ee3b27" } modify_external_catalogs: true
I20250115 14:52:16.012809 1751 tablet_replica.cc:331] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.012990 1751 raft_consensus.cc:2238] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.013119 1751 raft_consensus.cc:2267] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.013599 1751 ts_tablet_manager.cc:1905] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.014307 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
I20250115 14:52:16.014532 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 5df0250d08214a9783c392071126ed4a (table default.Table_90c834d5ef634e6aa797dd220e3d5faa [id=8ffbe34c2a11417f8ca50dbdd627bf22]). Expected version 2 got 0
I20250115 14:52:16.014664 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet ad988e25800f4af9a069b5a80582f749 (table default.Table_162eb94aa1674d21ae644021401a54e8 [id=e3887aa962874cfaa7ee89503ca1a49a]). Expected version 1 got 0
I20250115 14:52:16.016047 1164 tablet_bootstrap.cc:492] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:16.016182 1164 ts_tablet_manager.cc:1397] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20250115 14:52:16.016968 451 catalog_manager.cc:5526] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:16.017911 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 246720dbb4ed4216ac87fb4a3df79e2c (table default.Table_daaa2755fe9d4f54ae6ff33ae452ec05 [id=8c0229e6f6884ea7be3417678bb27377]). Expected version 1 got 0
I20250115 14:52:16.018151 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 5df0250d08214a9783c392071126ed4a (table default.Table_90c834d5ef634e6aa797dd220e3d5faa [id=8ffbe34c2a11417f8ca50dbdd627bf22]). Expected version 2 got 0
I20250115 14:52:16.018126 743 tablet_service.cc:1467] Processing CreateTablet for tablet 9f002e821f11402d9d2e9e34f23e0fa5 (DEFAULT_TABLE table=default.Table_c0e6fbc56b8545359a96da934e8410ba [id=b5e7d3ac61b64d13a1c1b59711357e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.018476 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f002e821f11402d9d2e9e34f23e0fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.019246 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1244) took 1352 ms. Trace:
I20250115 14:52:16.019344 1167 rpcz_store.cc:276] 0115 14:52:14.667131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.869806 (+202675us) service_pool.cc:225] Handling call
0115 14:52:16.019120 (+1149314us) ts_tablet_manager.cc:1145] Deleting tablet abefc717e0eb41b18725ef0c73154fdd
0115 14:52:16.019130 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.019226 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1149190}
I20250115 14:52:16.019627 1167 tablet_replica.cc:331] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:16.019817 1167 raft_consensus.cc:2238] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:16.020153 1167 raft_consensus.cc:2267] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.021524 1167 ts_tablet_manager.cc:1905] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.024896 877 tablet_service.cc:1467] Processing CreateTablet for tablet e4e3ad88d9de4077835df6c06db472c8 (DEFAULT_TABLE table=default.Table_9480f0c225c34cef9d44fc404aa3cefd [id=993313229db244f7b7a05166e7ece1c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.025715 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4e3ad88d9de4077835df6c06db472c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.026954 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_162eb94aa1674d21ae644021401a54e8 [id=e3887aa962874cfaa7ee89503ca1a49a] alter complete (version 1)
I20250115 14:52:16.030494 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2923dd60927d49f5b0ca09438cfd4ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.031016 893 tablet_service.cc:1514] Processing DeleteTablet for tablet a5018e358d234ea28023da76a8c61a86 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.031755 893 tablet_service.cc:1467] Processing CreateTablet for tablet 964a7d57633841dcb04ed07a25de4ad1 (DEFAULT_TABLE table=default.Table_bdb6b9f92fac4cb49ce119752a15b45e [id=0c33cd51b93a430ba33b24df0a0f6719]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.031939 2747 raft_consensus.cc:491] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.032102 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 964a7d57633841dcb04ed07a25de4ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.032043 2747 raft_consensus.cc:513] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.032637 2747 leader_election.cc:290] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.034819 1016 tablet_service.cc:1467] Processing CreateTablet for tablet bf86a1cd1d7b40ee8c3638af06770b79 (DEFAULT_TABLE table=default.Table_7423aeb75aaf4a9f8d98517868158323 [id=3f1e905acb9148d39fb7b36467184a3f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:16.036196 847 leader_election.cc:343] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 8f8bae9867d745afb1dfdf47bcb881da
I20250115 14:52:16.036384 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6eacb0c16f4c4ca8db8eb8f65be1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.037457 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf86a1cd1d7b40ee8c3638af06770b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.037566 2403 raft_consensus.cc:491] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.037705 2403 raft_consensus.cc:513] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.038349 2403 leader_election.cc:290] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:16.038909 710 leader_election.cc:343] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: a2fe51b51994491ab38a644bfc55e3ac
W20250115 14:52:16.039126 710 leader_election.cc:343] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: a2fe51b51994491ab38a644bfc55e3ac
I20250115 14:52:16.039214 710 leader_election.cc:304] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:16.039449 2403 raft_consensus.cc:2743] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.042533 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 9f002e821f11402d9d2e9e34f23e0fa5 (DEFAULT_TABLE table=default.Table_c0e6fbc56b8545359a96da934e8410ba [id=b5e7d3ac61b64d13a1c1b59711357e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.042892 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f002e821f11402d9d2e9e34f23e0fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.043040 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c519f995c4c413c85be48c4577d7db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.043534 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bbdf9c664434c4ba24deef67299c018. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.044199 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcff7017c77487d9ed5fd16f360f4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.045528 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_aac511b008a343a4a9b676f06ef31a5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.046070 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aac511b008a343a4a9b676f06ef31a5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.046285 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea876c00b5f441ba62ced9bb67f3818 (DEFAULT_TABLE table=default.Table_52240b9928824856bd6ce7d2ea06d625 [id=41b963c2c7e14f0fb996bd1de6251e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.046573 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea876c00b5f441ba62ced9bb67f3818. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.047195 1024 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c with delete_type TABLET_DATA_DELETED (replaced by tablet 27fc79ad43f14002a6fe1502a4347ea7) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.047533 1164 raft_consensus.cc:357] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.049266 1164 raft_consensus.cc:383] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.051218 883 tablet_service.cc:1514] Processing DeleteTablet for tablet a3b0bb374dc54533b7e8a5e5369c2f39 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.052209 452 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_1365cebf9380460289484fcc7b729f58 [id=855145809143431d81790dd36147a77b] alter complete (version 3)
I20250115 14:52:16.055780 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_da2d5c7907c24047bd4f7c0f6b70aa45" } modify_external_catalogs: true
I20250115 14:52:16.053606 1164 raft_consensus.cc:738] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.053824 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e8d3a3b353b14d8bb1f1c0afb200b824"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:16.056856 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 1298fdd0217e4df6bfbafc49b2f2c65c with delete_type TABLET_DATA_DELETED (replaced by tablet 0fb0195969e44d8d9d770c8b0d9c85bf) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:16.057000 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8d3a3b353b14d8bb1f1c0afb200b824 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.057641 743 tablet_service.cc:1514] Processing DeleteTablet for tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.057842 1166 tablet_replica.cc:331] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.058022 1166 raft_consensus.cc:2238] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.058169 1166 raft_consensus.cc:2267] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.058578 743 tablet_service.cc:1514] Processing DeleteTablet for tablet a2fe51b51994491ab38a644bfc55e3ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.059113 1166 ts_tablet_manager.cc:1905] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.057571 1164 consensus_queue.cc:260] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.064821 2747 raft_consensus.cc:491] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.064960 2747 raft_consensus.cc:513] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.065547 2747 leader_election.cc:290] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.067759 882 tablet_service.cc:1467] Processing CreateTablet for tablet ca04254f3b8846d490e5277f3d995069 (DEFAULT_TABLE table=default.Table_58d9a1abb0be4eaf8ebc5c55369a0154 [id=f28c9084fd1247d3be5dfef058d79a16]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.068166 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca04254f3b8846d490e5277f3d995069. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.068676 890 tablet_service.cc:1467] Processing CreateTablet for tablet edff9d5a740d4d5d8df02f4549c30700 (DEFAULT_TABLE table=default.Table_7bb0ca02228940ee91c77404cb564a29 [id=8f9b630f463949249cbb6088dfdc2bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.069010 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edff9d5a740d4d5d8df02f4549c30700. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.072400 881 tablet_service.cc:1514] Processing DeleteTablet for tablet b7c6c8d5c58142c0af3e726ea57207b9 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c33907919374a6fbf9e7175c7c9a485) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.074600 890 tablet_service.cc:1467] Processing CreateTablet for tablet 77863c5591e64a38926dcbd64c46437d (DEFAULT_TABLE table=default.Table_9be6523788f347129832e1de3230c4c7 [id=3cc080a5cca04dc285ce4dd4122faa8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.074929 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77863c5591e64a38926dcbd64c46437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.065587 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 5265d5eeac83481ca97801e3475886da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.065626 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e616d78933e4f84b9ec2acc51450b18 with delete_type TABLET_DATA_DELETED (replaced by tablet ba33eb3f5a40450db5cf115fbe8a17e7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.065177 1164 ts_tablet_manager.cc:1428] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.048s user 0.004s sys 0.000s
I20250115 14:52:16.065739 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 70425c25919b4d0b999450210e1cb33f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.068447 1166 ts_tablet_manager.cc:1918] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:16.076937 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1298fdd0217e4df6bfbafc49b2f2c65c because the tablet was not found. No further retry: Not found: Tablet not found: 1298fdd0217e4df6bfbafc49b2f2c65c
I20250115 14:52:16.079360 1164 tablet_bootstrap.cc:492] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:16.079854 1166 log.cc:1198] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/a9f0ecfceecf4acdbf2c2fbec88ff3cb
I20250115 14:52:16.081107 883 tablet_service.cc:1467] Processing CreateTablet for tablet 7dffe3941dcd409cbbd64efefe5831ba (DEFAULT_TABLE table=default.Table_e25f23f07b65483aa5d163d8017a19f1 [id=887a9cbe87684db5a474efcdbf3c2d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.082340 2563 raft_consensus.cc:491] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.082470 2563 raft_consensus.cc:513] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:16.082525 847 leader_election.cc:343] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 2e4c2707915e4e97ad6c66cb249c6d9e
I20250115 14:52:16.083016 2563 leader_election.cc:290] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.083653 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e97279185541b992340c24ede01656" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:16.083897 910 raft_consensus.cc:2463] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:16.084476 983 leader_election.cc:304] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.084812 2563 raft_consensus.cc:2798] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:16.084888 2563 raft_consensus.cc:491] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:16.084944 2563 raft_consensus.cc:3054] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.085681 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e97279185541b992340c24ede01656" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.086238 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dffe3941dcd409cbbd64efefe5831ba. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.086308 984 leader_election.cc:343] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.086136 1166 ts_tablet_manager.cc:1939] T a9f0ecfceecf4acdbf2c2fbec88ff3cb P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
W20250115 14:52:16.090968 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc741c3e00b543f99cf68122f58fd111 consensus state has no leader
I20250115 14:52:16.091320 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6ab2a32317d84ef0bc98f61078e58387" } modify_external_catalogs: true
I20250115 14:52:16.091492 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6ab2a32317d84ef0bc98f61078e58387" } modify_external_catalogs: true
I20250115 14:52:16.091985 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet bf648bb656634def8189fb514a07e144 (table default.Table_fd84fd3d47974e05a580fae352838c20 [id=a00115ecf6954f83b3118654f98e629d]). Expected version 1 got 0
I20250115 14:52:16.093089 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_740d21d5d42e459b8325b4cfaa1bf85f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:16.093286 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 7c3393ed28d242838d6f2f170184f7e8 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:16.094609 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6
I20250115 14:52:16.099728 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 87026b6faf75437eac21dedab1f41ed4
I20250115 14:52:16.100968 1025 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.101233 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_da2d5c7907c24047bd4f7c0f6b70aa45" } modify_external_catalogs: true
I20250115 14:52:16.103582 1022 tablet_service.cc:1514] Processing DeleteTablet for tablet 87026b6faf75437eac21dedab1f41ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.104640 1166 tablet_replica.cc:331] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.105237 1166 raft_consensus.cc:2238] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.105643 1166 raft_consensus.cc:2267] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.106320 1166 ts_tablet_manager.cc:1905] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.115111 1166 ts_tablet_manager.cc:1918] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:16.118304 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_740d21d5d42e459b8325b4cfaa1bf85f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.119128 881 tablet_service.cc:1514] Processing DeleteTablet for tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.119462 880 tablet_service.cc:1467] Processing CreateTablet for tablet 0bbdf9c664434c4ba24deef67299c018 (DEFAULT_TABLE table=default.Table_67c82d5293bd4157bc5887a9d6410293 [id=efa1ad976e3b4a22ba8866ebf80b6f4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.119462 884 tablet_service.cc:1467] Processing CreateTablet for tablet 2c519f995c4c413c85be48c4577d7db2 (DEFAULT_TABLE table=default.Table_adcc45a64d8840fba4e22623a802abb6 [id=aded3d5cc1c64969bc8528c9b5e038af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.119807 1166 log.cc:1198] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/a2fe51b51994491ab38a644bfc55e3ac
I20250115 14:52:16.119879 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c519f995c4c413c85be48c4577d7db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.120047 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bbdf9c664434c4ba24deef67299c018. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.120151 1166 ts_tablet_manager.cc:1939] T a2fe51b51994491ab38a644bfc55e3ac P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:16.122802 1166 tablet_replica.cc:331] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.123013 1166 raft_consensus.cc:2238] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.123159 1166 raft_consensus.cc:2267] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.124516 1166 ts_tablet_manager.cc:1905] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.126011 882 tablet_service.cc:1467] Processing CreateTablet for tablet 082920cb7a7f4994af352d59678518a4 (DEFAULT_TABLE table=default.Table_095871dcaa2547f8a7d859237e9bf4fa [id=5be54673655e4263b5f3fd9fe48b06a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.126392 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 082920cb7a7f4994af352d59678518a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.127230 880 tablet_service.cc:1467] Processing CreateTablet for tablet 2fcff7017c77487d9ed5fd16f360f4f9 (DEFAULT_TABLE table=default.Table_f7d31cfc0fe54ffdb1b9fc0b8b26fe3e [id=dbeb5e86f26349709ba1761e7eb7d8a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.127617 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcff7017c77487d9ed5fd16f360f4f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.133733 881 tablet_service.cc:1467] Processing CreateTablet for tablet 6ea876c00b5f441ba62ced9bb67f3818 (DEFAULT_TABLE table=default.Table_52240b9928824856bd6ce7d2ea06d625 [id=41b963c2c7e14f0fb996bd1de6251e96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.134079 874 tablet_service.cc:1514] Processing DeleteTablet for tablet 36de948083a946eba1a96f276e4f4f91 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e33ebec8b64f2398a7f1003b15cee7) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.134317 878 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:16.134369 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9ad3cda1056349a789a1751c5d386d0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ad3cda1056349a789a1751c5d386d0c already in progress: creating tablet
I20250115 14:52:16.134446 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8bae9867d745afb1dfdf47bcb881da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.134282 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 87026b6faf75437eac21dedab1f41ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.134274 892 tablet_service.cc:1467] Processing CreateTablet for tablet 9f002e821f11402d9d2e9e34f23e0fa5 (DEFAULT_TABLE table=default.Table_c0e6fbc56b8545359a96da934e8410ba [id=b5e7d3ac61b64d13a1c1b59711357e0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.134080 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ea876c00b5f441ba62ced9bb67f3818. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.139429 1167 ts_tablet_manager.cc:1918] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:16.139533 1167 log.cc:1198] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/1298fdd0217e4df6bfbafc49b2f2c65c
I20250115 14:52:16.139887 1167 ts_tablet_manager.cc:1939] T 1298fdd0217e4df6bfbafc49b2f2c65c P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:16.140390 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f002e821f11402d9d2e9e34f23e0fa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.149727 893 tablet_service.cc:1467] Processing CreateTablet for tablet 4adad3b609e8419890d463a66edc2189 (DEFAULT_TABLE table=default.Table_5392236e7689487c8c5909b627a06a12 [id=1f8a0d356ddd486e8faf98b98854afeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.152228 884 tablet_service.cc:1467] Processing CreateTablet for tablet bf86a1cd1d7b40ee8c3638af06770b79 (DEFAULT_TABLE table=default.Table_7423aeb75aaf4a9f8d98517868158323 [id=3f1e905acb9148d39fb7b36467184a3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.152510 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4adad3b609e8419890d463a66edc2189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.153788 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf86a1cd1d7b40ee8c3638af06770b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.158425 890 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6eacb0c16f4c4ca8db8eb8f65be1ae (DEFAULT_TABLE table=default.Table_cf227e000a9a4a8f8565e985bb801c61 [id=198fa58e5e3f4e87bcf297e5f1c69fca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.158838 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6eacb0c16f4c4ca8db8eb8f65be1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.164201 2747 consensus_queue.cc:1035] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:16.164897 880 tablet_service.cc:1467] Processing CreateTablet for tablet adcb42acbc7142498155fdb8fd04fb8b (DEFAULT_TABLE table=default.Table_883692e629d34120b90cdbf0744cdfc8 [id=e95f9b6c227c4903b523486a033650e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.165355 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adcb42acbc7142498155fdb8fd04fb8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.167263 1168 tablet_bootstrap.cc:492] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:16.167448 1168 ts_tablet_manager.cc:1397] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.250s user 0.003s sys 0.006s
I20250115 14:52:16.168255 1168 raft_consensus.cc:357] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.168617 1168 raft_consensus.cc:383] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.168709 1168 raft_consensus.cc:738] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.173481 2403 raft_consensus.cc:491] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.173643 2403 raft_consensus.cc:513] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.174296 2403 leader_election.cc:290] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:16.175429 710 leader_election.cc:343] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 6e616d78933e4f84b9ec2acc51450b18
W20250115 14:52:16.186170 710 leader_election.cc:343] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6e616d78933e4f84b9ec2acc51450b18
I20250115 14:52:16.186327 710 leader_election.cc:304] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:16.186862 2403 raft_consensus.cc:2743] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.188566 1168 consensus_queue.cc:260] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.189177 1168 ts_tablet_manager.cc:1428] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20250115 14:52:16.189478 1168 tablet_bootstrap.cc:492] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:16.191255 1168 tablet_bootstrap.cc:654] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:16.195530 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 13a4837b72eb4008ad612a9fc86dfac7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13a4837b72eb4008ad612a9fc86dfac7 already in progress: creating tablet
W20250115 14:52:16.196270 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 24729156769443f8926577bae7707847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24729156769443f8926577bae7707847 already in progress: creating tablet
W20250115 14:52:16.196666 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7830c683a109448ab06c15b5de692e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7830c683a109448ab06c15b5de692e8b already in progress: creating tablet
I20250115 14:52:16.197736 750 tablet_service.cc:1514] Processing DeleteTablet for tablet b76d1bbf90c6416ca60805c9d8053bd4 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e10fe1445be457a8c848c30771af866) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.197995 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 87026b6faf75437eac21dedab1f41ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:16.198051 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 3e834623f9c7482d965afd23c89a2510 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e834623f9c7482d965afd23c89a2510 already in progress: creating tablet
W20250115 14:52:16.198323 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet abefc717e0eb41b18725ef0c73154fdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abefc717e0eb41b18725ef0c73154fdd already in progress: creating tablet
I20250115 14:52:16.198489 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:15 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.199023 768 raft_consensus.cc:3054] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.200838 2747 raft_consensus.cc:491] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.200963 2747 raft_consensus.cc:513] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.201501 2747 leader_election.cc:290] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:16.203933 848 leader_election.cc:343] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:16.204257 848 leader_election.cc:343] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.204248 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8bae9867d745afb1dfdf47bcb881da" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.204373 848 leader_election.cc:304] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.204568 2657 raft_consensus.cc:2743] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:16.204687 848 leader_election.cc:343] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.204945 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e4c2707915e4e97ad6c66cb249c6d9e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:16.204946 848 consensus_peers.cc:487] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:16.206408 1164 tablet_bootstrap.cc:654] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.206607 1028 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43552 (request call id 1291) took 1027 ms. Trace:
I20250115 14:52:16.206693 1028 rpcz_store.cc:276] 0115 14:52:15.178680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.181192 (+ 2512us) service_pool.cc:225] Handling call
0115 14:52:15.181506 (+ 314us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0115 14:52:15.181526 (+ 20us) ts_tablet_manager.cc:676] Creating new metadata...
0115 14:52:16.095261 (+913735us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.206438 (+111177us) tablet_replica.cc:206] Creating consensus instance
0115 14:52:16.206587 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 14:52:16.214473 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet a9f0ecfceecf4acdbf2c2fbec88ff3cb (table default.Table_1c51a5aaced84915a1dba864731460fe [id=ec452771986240f4a7654505a30d3d9d]) successfully deleted
I20250115 14:52:16.214665 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet a2fe51b51994491ab38a644bfc55e3ac (table default.Table_9f7b9d2bd72741fbb3132278c8a77d1c [id=15f2a471b89449459d7dbf804b4e922c]) successfully deleted
I20250115 14:52:16.218991 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 246720dbb4ed4216ac87fb4a3df79e2c (table default.Table_daaa2755fe9d4f54ae6ff33ae452ec05 [id=8c0229e6f6884ea7be3417678bb27377]). Expected version 1 got 0
I20250115 14:52:16.221000 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
I20250115 14:52:16.219622 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ebc003502c644be98ae8fec069e7bb39" } new_table_name: "default.Table_b578368ee37e47b6a6dc45444cf2ee8e" modify_external_catalogs: true
I20250115 14:52:16.219313 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_40404fbf248d43aeab53e7db74beb0c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.222649 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40404fbf248d43aeab53e7db74beb0c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.223446 1017 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43552 (request call id 1288) took 1047 ms. Trace:
I20250115 14:52:16.223529 1017 rpcz_store.cc:276] 0115 14:52:15.175508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.175715 (+ 207us) service_pool.cc:225] Handling call
0115 14:52:15.176333 (+ 618us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0115 14:52:15.176355 (+ 22us) ts_tablet_manager.cc:676] Creating new metadata...
0115 14:52:16.099748 (+923393us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.223258 (+123510us) tablet_replica.cc:206] Creating consensus instance
0115 14:52:16.223427 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250115 14:52:16.223672 442 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 16a88475f9ff42c1952e5a14ec852e90 (table default.Table_f4904239f07b44888c5b08324a52aa5b [id=3c1f2f64fe374b5294ef6130f252840f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 16a88475f9ff42c1952e5a14ec852e90 consensus state has no leader
I20250115 14:52:16.224874 448 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet bf648bb656634def8189fb514a07e144 (table default.Table_fd84fd3d47974e05a580fae352838c20 [id=a00115ecf6954f83b3118654f98e629d]). Expected version 1 got 0
I20250115 14:52:16.226786 1162 tablet_bootstrap.cc:492] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.227057 1162 ts_tablet_manager.cc:1397] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.238s user 0.008s sys 0.009s
I20250115 14:52:16.227135 448 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_736cec83be284849af7e07aa9d729965" } modify_external_catalogs: true
I20250115 14:52:16.227299 448 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_736cec83be284849af7e07aa9d729965" } modify_external_catalogs: true
I20250115 14:52:16.227826 1023 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43552 (request call id 1295) took 1044 ms. Trace:
I20250115 14:52:16.227934 1023 rpcz_store.cc:276] 0115 14:52:15.183483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.183639 (+ 156us) service_pool.cc:225] Handling call
0115 14:52:15.183935 (+ 296us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0115 14:52:15.183956 (+ 21us) ts_tablet_manager.cc:676] Creating new metadata...
0115 14:52:16.220468 (+1036512us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.227678 (+ 7210us) tablet_replica.cc:206] Creating consensus instance
0115 14:52:16.227808 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250115 14:52:16.227829 1162 raft_consensus.cc:357] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.229938 1162 raft_consensus.cc:383] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.230195 1162 raft_consensus.cc:738] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.230516 1162 consensus_queue.cc:260] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.231598 2748 consensus_queue.cc:1035] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:52:16.232172 848 consensus_peers.cc:487] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d13b4c0edc2c4fc9a1c527e1ff6a3d00. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:16.232547 1166 ts_tablet_manager.cc:1918] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.232685 1166 log.cc:1198] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/5265d5eeac83481ca97801e3475886da
I20250115 14:52:16.232710 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1245) took 1565 ms. Trace:
I20250115 14:52:16.232817 1167 rpcz_store.cc:276] 0115 14:52:14.667191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.869935 (+202744us) service_pool.cc:225] Handling call
0115 14:52:16.019584 (+1149649us) ts_tablet_manager.cc:1145] Deleting tablet 1298fdd0217e4df6bfbafc49b2f2c65c
0115 14:52:16.019593 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.136642 (+117049us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.139359 (+ 2717us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.232685 (+ 93326us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":623,"tablet-delete.queue_time_us":1149561}
I20250115 14:52:16.233011 1166 ts_tablet_manager.cc:1939] T 5265d5eeac83481ca97801e3475886da P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:16.233162 1167 tablet_replica.cc:331] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:16.233350 1167 raft_consensus.cc:2238] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.233484 1167 raft_consensus.cc:2267] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.233989 1167 ts_tablet_manager.cc:1905] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:16.234938 848 leader_election.cc:343] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.235049 848 leader_election.cc:304] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.235248 2748 raft_consensus.cc:2743] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:16.235405 848 leader_election.cc:343] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.235494 848 leader_election.cc:304] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.235657 2748 raft_consensus.cc:2743] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.241080 1166 tablet_replica.cc:331] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.241842 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a79f880415a4b16aabdbb7cd989efdb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.243395 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 1298fdd0217e4df6bfbafc49b2f2c65c (table default.Table_15773744107143d6b1e5215c832f0a3f [id=f4305d1bfe12463fa68fb1405be5267f]) successfully deleted
I20250115 14:52:16.245533 1168 tablet_bootstrap.cc:492] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:16.245664 1168 ts_tablet_manager.cc:1397] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.000s
I20250115 14:52:16.244153 2563 raft_consensus.cc:513] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.247130 2563 leader_election.cc:290] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.247432 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e97279185541b992340c24ede01656" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:16.247623 911 raft_consensus.cc:3054] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.249653 1166 raft_consensus.cc:2238] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.249821 1166 raft_consensus.cc:2267] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.250240 911 raft_consensus.cc:2463] T a5e97279185541b992340c24ede01656 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:16.250327 1166 ts_tablet_manager.cc:1905] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.250751 983 leader_election.cc:304] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.252112 1167 ts_tablet_manager.cc:1918] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.252215 1167 log.cc:1198] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/1140016d11a74a2cadba494c0b319c54
I20250115 14:52:16.253062 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 5265d5eeac83481ca97801e3475886da (table default.Table_4add05103df84280812bf61910d76d1d [id=47db9a9013ec4a2cb3cc0cc8f9caba12]) successfully deleted
I20250115 14:52:16.254169 2829 raft_consensus.cc:2798] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:16.254343 2829 raft_consensus.cc:695] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:16.254621 2829 consensus_queue.cc:237] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.254823 2835 raft_consensus.cc:491] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.254930 2835 raft_consensus.cc:513] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.255313 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b3daf6416ad847aeaa83087cc81c9239
I20250115 14:52:16.255534 2835 leader_election.cc:290] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.255970 1048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a79f880415a4b16aabdbb7cd989efdb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:16.256182 1048 raft_consensus.cc:2463] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:16.257277 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04daad461f0b433ba635df87439b5bc1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:16.257428 1042 raft_consensus.cc:2463] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:16.257437 2836 consensus_queue.cc:1035] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:16.257668 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_214a95027bd94905b10b5c7285a31b00" } new_table_name: "default.Table_63e98809325344f4bed871f7974db0e8" modify_external_catalogs: true
I20250115 14:52:16.258179 847 leader_election.cc:304] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.258499 2747 raft_consensus.cc:2798] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:16.258561 2747 raft_consensus.cc:491] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:16.258622 2747 raft_consensus.cc:3054] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.261174 2747 raft_consensus.cc:513] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.261348 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet f400880835ab4efdb2af0be5553d20dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.261758 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet b3daf6416ad847aeaa83087cc81c9239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.261766 2747 leader_election.cc:290] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.263242 847 leader_election.cc:304] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.263535 2747 raft_consensus.cc:2798] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:16.263633 2747 raft_consensus.cc:491] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:16.263731 2747 raft_consensus.cc:3054] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.266449 2747 raft_consensus.cc:513] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.267179 2747 leader_election.cc:290] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.267702 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a79f880415a4b16aabdbb7cd989efdb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:16.267867 1042 raft_consensus.cc:3054] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.269284 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e97279185541b992340c24ede01656" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:16.270576 1042 raft_consensus.cc:2463] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
W20250115 14:52:16.271569 984 leader_election.cc:343] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:16.272267 984 consensus_peers.cc:487] T 246720dbb4ed4216ac87fb4a3df79e2c P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:16.272897 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04daad461f0b433ba635df87439b5bc1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:16.273079 1042 raft_consensus.cc:3054] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.276014 1042 raft_consensus.cc:2463] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:16.278023 847 leader_election.cc:304] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.278370 2835 raft_consensus.cc:2798] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:16.278522 2835 raft_consensus.cc:695] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:16.278656 1751 ts_tablet_manager.cc:1918] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.278791 1751 log.cc:1198] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4f89ec0a5d68408c897f43ab39df62b1
I20250115 14:52:16.278808 2835 consensus_queue.cc:237] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.279191 1751 ts_tablet_manager.cc:1939] T 4f89ec0a5d68408c897f43ab39df62b1 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.279922 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1194) took 2683 ms. Trace:
I20250115 14:52:16.280056 847 leader_election.cc:304] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.280586 2748 raft_consensus.cc:2798] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:16.280686 2748 raft_consensus.cc:695] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:16.280896 2748 consensus_queue.cc:237] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.282003 1164 tablet_bootstrap.cc:492] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:16.282145 1164 ts_tablet_manager.cc:1397] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.204s user 0.003s sys 0.007s
I20250115 14:52:16.282861 1164 raft_consensus.cc:357] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.283049 1164 raft_consensus.cc:383] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.283097 1164 raft_consensus.cc:738] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.283280 1164 consensus_queue.cc:260] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.283763 1164 ts_tablet_manager.cc:1428] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:16.284051 1164 tablet_bootstrap.cc:492] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:16.292200 1166 ts_tablet_manager.cc:1918] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.292359 1166 log.cc:1198] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/6e616d78933e4f84b9ec2acc51450b18
I20250115 14:52:16.292665 1166 ts_tablet_manager.cc:1939] T 6e616d78933e4f84b9ec2acc51450b18 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:16.293175 1164 tablet_bootstrap.cc:654] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.294389 1751 rpcz_store.cc:276] 0115 14:52:13.596648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.596702 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:16.012759 (+2416057us) ts_tablet_manager.cc:1145] Deleting tablet 4f89ec0a5d68408c897f43ab39df62b1
0115 14:52:16.012768 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.174823 (+162055us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.278547 (+103724us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.279903 (+ 1356us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2415878}
I20250115 14:52:16.280853 1162 ts_tablet_manager.cc:1428] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.054s user 0.001s sys 0.004s
I20250115 14:52:16.295054 1162 tablet_bootstrap.cc:492] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.296489 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 4f89ec0a5d68408c897f43ab39df62b1 (table default.Table_d1198c17efa04e4ea060b6d89580342a [id=5e381e3344a34b53a930fcdfc8c8fa25]) successfully deleted
I20250115 14:52:16.296784 1162 tablet_bootstrap.cc:654] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.297753 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1195) took 2656 ms. Trace:
I20250115 14:52:16.297848 1751 rpcz_store.cc:276] 0115 14:52:13.641805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.641865 (+ 60us) service_pool.cc:225] Handling call
0115 14:52:16.294762 (+2652897us) ts_tablet_manager.cc:1145] Deleting tablet 0f82d944a7e04ace8e4addb3897d2c8e
0115 14:52:16.297615 (+ 2853us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.297735 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2652707}
I20250115 14:52:16.298413 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1196) took 2656 ms. Trace:
I20250115 14:52:16.298478 1751 rpcz_store.cc:276] 0115 14:52:13.642409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.642458 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:16.298305 (+2655847us) ts_tablet_manager.cc:1145] Deleting tablet 23318b1ad0774a76880def17a64ac059
0115 14:52:16.298314 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.298396 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2655695}
I20250115 14:52:16.298987 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1197) took 2655 ms. Trace:
W20250115 14:52:16.299108 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 0f82d944a7e04ace8e4addb3897d2c8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f82d944a7e04ace8e4addb3897d2c8e already in progress: creating tablet
I20250115 14:52:16.299187 2774 tablet.cc:1721] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250115 14:52:16.299372 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 23318b1ad0774a76880def17a64ac059 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23318b1ad0774a76880def17a64ac059 already in progress: creating tablet
I20250115 14:52:16.299052 1751 rpcz_store.cc:276] 0115 14:52:13.643163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.643205 (+ 42us) service_pool.cc:225] Handling call
0115 14:52:16.298759 (+2655554us) ts_tablet_manager.cc:1145] Deleting tablet 356a0dfd7bb746548e0fb5b6a0935318
0115 14:52:16.298767 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.298843 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2655427}
I20250115 14:52:16.299921 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1199) took 2626 ms. Trace:
W20250115 14:52:16.300021 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 356a0dfd7bb746548e0fb5b6a0935318 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 356a0dfd7bb746548e0fb5b6a0935318 already in progress: creating tablet
I20250115 14:52:16.299997 1751 rpcz_store.cc:276] 0115 14:52:13.673877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.673929 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:16.299827 (+2625898us) ts_tablet_manager.cc:1145] Deleting tablet 50f8d93d655a447cb9e92997342a61a7
0115 14:52:16.299836 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.299906 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2625724}
W20250115 14:52:16.300482 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 50f8d93d655a447cb9e92997342a61a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50f8d93d655a447cb9e92997342a61a7 already in progress: creating tablet
I20250115 14:52:16.304613 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1200) took 2601 ms. Trace:
I20250115 14:52:16.305303 1751 rpcz_store.cc:276] 0115 14:52:13.703255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.703310 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:16.304507 (+2601197us) ts_tablet_manager.cc:1145] Deleting tablet 93aaea90b0924327807e7f594154b14e
0115 14:52:16.304515 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.304597 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2601018}
W20250115 14:52:16.306077 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 93aaea90b0924327807e7f594154b14e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93aaea90b0924327807e7f594154b14e already in progress: creating tablet
I20250115 14:52:16.306921 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1201) took 2568 ms. Trace:
I20250115 14:52:16.307186 1751 rpcz_store.cc:276] 0115 14:52:13.738797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.738854 (+ 57us) service_pool.cc:225] Handling call
0115 14:52:16.306468 (+2567614us) ts_tablet_manager.cc:1145] Deleting tablet ec9e7c8e59aa4ea59bad35c8a7591a79
0115 14:52:16.306829 (+ 361us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.306906 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2567397}
W20250115 14:52:16.307896 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ec9e7c8e59aa4ea59bad35c8a7591a79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec9e7c8e59aa4ea59bad35c8a7591a79 already in progress: creating tablet
I20250115 14:52:16.308722 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1202) took 2568 ms. Trace:
I20250115 14:52:16.309587 1164 tablet_bootstrap.cc:492] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:16.309690 1751 rpcz_store.cc:276] 0115 14:52:13.739781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.739830 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:16.308226 (+2568396us) ts_tablet_manager.cc:1145] Deleting tablet 7cf17435b88c4e87800daefcd4fa35ec
0115 14:52:16.308235 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.308705 (+ 470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2568245}
W20250115 14:52:16.310113 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7cf17435b88c4e87800daefcd4fa35ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7cf17435b88c4e87800daefcd4fa35ec already in progress: creating tablet
I20250115 14:52:16.310528 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1205) took 2514 ms. Trace:
I20250115 14:52:16.310714 1166 tablet_replica.cc:331] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.310848 1751 rpcz_store.cc:276] 0115 14:52:13.795965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.796029 (+ 64us) service_pool.cc:225] Handling call
0115 14:52:16.310387 (+2514358us) ts_tablet_manager.cc:1145] Deleting tablet abe3c66176664fa99c7a1fb480b56cca
0115 14:52:16.310400 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.310507 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2514183}
I20250115 14:52:16.310783 1164 ts_tablet_manager.cc:1397] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250115 14:52:16.310952 1166 raft_consensus.cc:2238] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.311822 1166 raft_consensus.cc:2267] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:16.312095 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet abe3c66176664fa99c7a1fb480b56cca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abe3c66176664fa99c7a1fb480b56cca already in progress: creating tablet
I20250115 14:52:16.312639 1751 tablet_replica.cc:331] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.312958 1751 raft_consensus.cc:2238] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.313007 2631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250115 14:52:16.313611 1751 raft_consensus.cc:2267] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.314139 2631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250115 14:52:16.314443 2631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250115 14:52:16.313930 1164 raft_consensus.cc:357] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.315150 1164 raft_consensus.cc:383] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.315502 1164 raft_consensus.cc:738] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.316402 2403 raft_consensus.cc:491] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.316514 2403 raft_consensus.cc:513] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.317077 2403 leader_election.cc:290] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.317894 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1140016d11a74a2cadba494c0b319c54" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:16.318445 710 leader_election.cc:343] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: Tablet not RUNNING: STOPPED
W20250115 14:52:16.319037 710 leader_election.cc:343] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 1140016d11a74a2cadba494c0b319c54
I20250115 14:52:16.318585 1751 ts_tablet_manager.cc:1905] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.319129 710 leader_election.cc:304] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:16.319525 2403 raft_consensus.cc:2743] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.318406 1164 consensus_queue.cc:260] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.320232 1166 ts_tablet_manager.cc:1905] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.321975 1164 ts_tablet_manager.cc:1428] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250115 14:52:16.324113 1164 tablet_bootstrap.cc:492] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:16.328939 1164 tablet_bootstrap.cc:654] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.330744 1168 raft_consensus.cc:357] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.330943 1168 raft_consensus.cc:383] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.330988 1168 raft_consensus.cc:738] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.331183 1168 consensus_queue.cc:260] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.332800 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04daad461f0b433ba635df87439b5bc1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:16.332991 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04daad461f0b433ba635df87439b5bc1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.333338 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a79f880415a4b16aabdbb7cd989efdb" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:16.335942 1166 ts_tablet_manager.cc:1918] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:16.336089 1166 log.cc:1198] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/70425c25919b4d0b999450210e1cb33f
I20250115 14:52:16.336061 1751 ts_tablet_manager.cc:1918] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.336165 1751 log.cc:1198] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/72a97d102c174bc3a5e3a3c5762033b3
I20250115 14:52:16.336498 1166 ts_tablet_manager.cc:1939] T 70425c25919b4d0b999450210e1cb33f P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:16.337528 1166 tablet_replica.cc:331] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.337723 1166 raft_consensus.cc:2238] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.337898 1166 raft_consensus.cc:2267] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.337528 1164 tablet_bootstrap.cc:492] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:16.338644 1166 ts_tablet_manager.cc:1905] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.343410 2849 tablet.cc:1721] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:16.344652 1166 ts_tablet_manager.cc:1918] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.344753 1166 log.cc:1198] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/b76d1bbf90c6416ca60805c9d8053bd4
I20250115 14:52:16.345005 1166 ts_tablet_manager.cc:1939] T b76d1bbf90c6416ca60805c9d8053bd4 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:16.346020 1164 ts_tablet_manager.cc:1397] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250115 14:52:16.347440 1164 raft_consensus.cc:357] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.347628 1164 raft_consensus.cc:383] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.347687 1164 raft_consensus.cc:738] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.347882 1164 consensus_queue.cc:260] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.348378 1164 ts_tablet_manager.cc:1428] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:16.348644 1164 tablet_bootstrap.cc:492] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:16.349752 1166 tablet_replica.cc:331] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.349929 1166 raft_consensus.cc:2238] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.350093 1166 raft_consensus.cc:2267] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.350569 1166 ts_tablet_manager.cc:1905] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.351487 1168 ts_tablet_manager.cc:1428] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.106s user 0.001s sys 0.001s
I20250115 14:52:16.351796 1168 tablet_bootstrap.cc:492] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:16.353406 1168 tablet_bootstrap.cc:654] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.356956 742 tablet_service.cc:1514] Processing DeleteTablet for tablet b3daf6416ad847aeaa83087cc81c9239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.357321 1751 ts_tablet_manager.cc:1939] T 72a97d102c174bc3a5e3a3c5762033b3 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.358640 1164 tablet_bootstrap.cc:654] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.362639 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1206) took 2563 ms. Trace:
I20250115 14:52:16.363023 1751 rpcz_store.cc:276] 0115 14:52:13.799122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.799173 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:16.312566 (+2513393us) ts_tablet_manager.cc:1145] Deleting tablet 72a97d102c174bc3a5e3a3c5762033b3
0115 14:52:16.312575 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.328399 (+ 15824us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.335969 (+ 7570us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.362619 (+ 26650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2513230}
I20250115 14:52:16.364427 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 72a97d102c174bc3a5e3a3c5762033b3 (table default.Table_0a8e879b253d48d3b02055cb2bd9663e [id=e07c949059424311bb6faf43de5dd443]) successfully deleted
I20250115 14:52:16.364858 1751 tablet_replica.cc:331] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.365229 1751 raft_consensus.cc:2238] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.365705 1751 raft_consensus.cc:2267] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.366458 1751 ts_tablet_manager.cc:1905] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.366854 1164 tablet_bootstrap.cc:492] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:16.366981 1164 ts_tablet_manager.cc:1397] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20250115 14:52:16.367463 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 6e616d78933e4f84b9ec2acc51450b18 (table default.Table_ff33645127a443efb2b776d808942dba [id=f8c4ca50220a4dbf9b9480213d836e5f]) successfully deleted
I20250115 14:52:16.367691 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 70425c25919b4d0b999450210e1cb33f (table default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]) successfully deleted
I20250115 14:52:16.367847 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet b76d1bbf90c6416ca60805c9d8053bd4 (table default.Table_9edd581e8fc4457697465360c4504ff1 [id=0ef6f07cf29e46e69bf2a4449187a4e7]) successfully deleted
W20250115 14:52:16.368041 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 87026b6faf75437eac21dedab1f41ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87026b6faf75437eac21dedab1f41ed4 already in progress: creating tablet
I20250115 14:52:16.371990 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7c3393ed28d242838d6f2f170184f7e8
I20250115 14:52:16.367668 1164 raft_consensus.cc:357] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.372316 1164 raft_consensus.cc:383] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.372385 1164 raft_consensus.cc:738] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.372462 450 catalog_manager.cc:6596] ReplaceTablet: tablet 7c3393ed28d242838d6f2f170184f7e8 deleted and replaced by tablet 9948d4a0552a4dfcad497d4fe1472fb1
I20250115 14:52:16.372567 1164 consensus_queue.cc:260] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.372828 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c3393ed28d242838d6f2f170184f7e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9948d4a0552a4dfcad497d4fe1472fb1) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.373030 1164 ts_tablet_manager.cc:1428] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250115 14:52:16.373282 1164 tablet_bootstrap.cc:492] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:16.373615 1162 tablet_bootstrap.cc:492] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.373719 1162 ts_tablet_manager.cc:1397] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.079s user 0.007s sys 0.007s
I20250115 14:52:16.374302 1162 raft_consensus.cc:357] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.374464 1162 raft_consensus.cc:383] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.374509 1162 raft_consensus.cc:738] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.374595 1164 tablet_bootstrap.cc:654] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.374747 450 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b (table default.Table_f634603ea42b4215b8ef3df8c446ada7 [id=174b82e1e06f4073a74a25aad8fc8045]). Expected version 2 got 0
I20250115 14:52:16.374711 1162 consensus_queue.cc:260] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.374950 450 catalog_manager.cc:5526] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:16.375320 450 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a5e97279185541b992340c24ede01656 (table default.Table_a8391687956e4259a71808f267cb4ff0 [id=361ed20953ab4e79b1112ae76b95a423]). Expected version 3 got 0
I20250115 14:52:16.376863 1162 ts_tablet_manager.cc:1428] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:16.377703 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4968ce56c6b4457097880a8ea7f924d8
I20250115 14:52:16.378341 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_139b3230a751497792db52e42d6fcd5f" } new_table_name: "default.Table_61919c493ca64b90818d51b3ee940d8b" modify_external_catalogs: true
I20250115 14:52:16.378556 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 4968ce56c6b4457097880a8ea7f924d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.378945 1162 tablet_bootstrap.cc:492] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.379753 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d6182b9f2b0e4105887762b69f004053" } new_table_name: "default.Table_d79f3054627d46fe87697c3985a729ed" modify_external_catalogs: true
I20250115 14:52:16.381387 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_25bedca633db44cfbf7fcc28e160d831"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.381868 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25bedca633db44cfbf7fcc28e160d831 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.383216 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_35647648e7e64b1382583fb35ae12dc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.383630 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35647648e7e64b1382583fb35ae12dc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.384496 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7db35ff0383549de86230bb1eb89389f
W20250115 14:52:16.356794 848 leader_election.cc:343] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.384997 448 master_service.cc:946] ReplaceTablet: received request to replace tablet f1e3879802464acaa279dfe4fbb634bb from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:16.385371 848 leader_election.cc:343] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.385522 1162 tablet_bootstrap.cc:654] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:16.385638 848 leader_election.cc:343] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:16.385879 848 leader_election.cc:343] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.356973 890 tablet_service.cc:1514] Processing DeleteTablet for tablet b3daf6416ad847aeaa83087cc81c9239 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.357981 2549 raft_consensus.cc:491] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.387548 2549 raft_consensus.cc:513] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.388154 2549 leader_election.cc:290] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.388717 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c04fa23e7c40a5b5e07852e5d74a27" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:16.388917 1046 raft_consensus.cc:2463] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:16.389257 768 raft_consensus.cc:3054] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.389449 847 leader_election.cc:304] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.389739 1166 ts_tablet_manager.cc:1918] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:16.389843 1166 log.cc:1198] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/3c2ee5dfdef14ec092a2f9d4bfd295a6
I20250115 14:52:16.390159 1166 ts_tablet_manager.cc:1939] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:16.390326 2549 raft_consensus.cc:2798] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:16.390388 2549 raft_consensus.cc:491] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:16.390444 2549 raft_consensus.cc:3054] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.392721 1751 ts_tablet_manager.cc:1918] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.392805 1751 log.cc:1198] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3f541e916d264738842cd965c7a2a5ec
I20250115 14:52:16.393062 1751 ts_tablet_manager.cc:1939] T 3f541e916d264738842cd965c7a2a5ec P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.393224 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 7db35ff0383549de86230bb1eb89389f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.393692 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1207) took 2542 ms. Trace:
I20250115 14:52:16.393757 1751 rpcz_store.cc:276] 0115 14:52:13.851359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.851458 (+ 99us) service_pool.cc:225] Handling call
0115 14:52:16.364799 (+2513341us) ts_tablet_manager.cc:1145] Deleting tablet 3f541e916d264738842cd965c7a2a5ec
0115 14:52:16.364809 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.388041 (+ 23232us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.392643 (+ 4602us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.393674 (+ 1031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2498500}
I20250115 14:52:16.394155 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3f541e916d264738842cd965c7a2a5ec (table default.Table_33a2bbb1ba7d426caf1eee40268cec01 [id=8d19600fb0354c028cafa03a1c87e5c9]) successfully deleted
I20250115 14:52:16.396667 1162 tablet_bootstrap.cc:492] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.396829 1162 ts_tablet_manager.cc:1397] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250115 14:52:16.397480 1162 raft_consensus.cc:357] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.397671 1162 raft_consensus.cc:383] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.397727 1162 raft_consensus.cc:738] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.397969 1162 consensus_queue.cc:260] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.401012 1162 ts_tablet_manager.cc:1428] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250115 14:52:16.401340 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c3393ed28d242838d6f2f170184f7e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9948d4a0552a4dfcad497d4fe1472fb1) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.401448 1162 tablet_bootstrap.cc:492] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.401757 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 4968ce56c6b4457097880a8ea7f924d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.402614 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 7db35ff0383549de86230bb1eb89389f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.403640 1162 tablet_bootstrap.cc:654] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.403937 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1208) took 2471 ms. Trace:
I20250115 14:52:16.404378 1751 rpcz_store.cc:276] 0115 14:52:13.932826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.932887 (+ 61us) service_pool.cc:225] Handling call
0115 14:52:16.403796 (+2470909us) ts_tablet_manager.cc:1145] Deleting tablet 11aeb6c73b4d42058736ffce551e311c
0115 14:52:16.403805 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.403922 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2470713}
W20250115 14:52:16.405421 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 11aeb6c73b4d42058736ffce551e311c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11aeb6c73b4d42058736ffce551e311c already in progress: creating tablet
I20250115 14:52:16.405754 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c04fa23e7c40a5b5e07852e5d74a27" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.405874 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1209) took 2463 ms. Trace:
I20250115 14:52:16.406035 1751 rpcz_store.cc:276] 0115 14:52:13.942226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.942286 (+ 60us) service_pool.cc:225] Handling call
0115 14:52:16.405741 (+2463455us) ts_tablet_manager.cc:1145] Deleting tablet eb78eb2c5b9b44fba1a4e140a8e102d7
0115 14:52:16.405750 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.405858 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2463275}
W20250115 14:52:16.406499 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet eb78eb2c5b9b44fba1a4e140a8e102d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb78eb2c5b9b44fba1a4e140a8e102d7 already in progress: creating tablet
I20250115 14:52:16.406905 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1210) took 2459 ms. Trace:
I20250115 14:52:16.407013 1751 rpcz_store.cc:276] 0115 14:52:13.947362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.947416 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:16.406789 (+2459373us) ts_tablet_manager.cc:1145] Deleting tablet 4e459144038d411d9e76322dee8f004b
0115 14:52:16.406798 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.406890 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2459202}
W20250115 14:52:16.407410 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 4e459144038d411d9e76322dee8f004b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e459144038d411d9e76322dee8f004b already in progress: creating tablet
I20250115 14:52:16.407752 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1211) took 2440 ms. Trace:
I20250115 14:52:16.407866 1751 rpcz_store.cc:276] 0115 14:52:13.967754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.967806 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:16.407659 (+2439853us) ts_tablet_manager.cc:1145] Deleting tablet d36fa3146f0f40a59e5f96cce203fee0
0115 14:52:16.407667 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.407737 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2439689}
W20250115 14:52:16.408237 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d36fa3146f0f40a59e5f96cce203fee0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d36fa3146f0f40a59e5f96cce203fee0 already in progress: creating tablet
I20250115 14:52:16.408587 1751 tablet_replica.cc:331] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.408833 1751 raft_consensus.cc:2238] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:16.409197 1751 raft_consensus.cc:2267] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.409754 1751 ts_tablet_manager.cc:1905] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:16.422538 848 leader_election.cc:343] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.424314 774 raft_consensus.cc:3054] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.425633 1751 ts_tablet_manager.cc:1918] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:16.425935 1751 log.cc:1198] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f06544890cbf460cbe218690e998eb1d
I20250115 14:52:16.426375 1751 ts_tablet_manager.cc:1939] T f06544890cbf460cbe218690e998eb1d P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.427467 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1212) took 2444 ms. Trace:
I20250115 14:52:16.429945 1751 rpcz_store.cc:276] 0115 14:52:13.982776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:13.983070 (+ 294us) service_pool.cc:225] Handling call
0115 14:52:16.408538 (+2425468us) ts_tablet_manager.cc:1145] Deleting tablet f06544890cbf460cbe218690e998eb1d
0115 14:52:16.408546 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.417793 (+ 9247us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.420843 (+ 3050us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.427448 (+ 6605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2425321}
W20250115 14:52:16.430101 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:40227, user_credentials={real_user=slave}} blocked reactor thread for 60960us
I20250115 14:52:16.430835 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f06544890cbf460cbe218690e998eb1d (table default.Table_1f1b7f31ef374e49add057beb69dbbcf [id=908287c385104ac0b9c3266b1b674671]) successfully deleted
I20250115 14:52:16.431303 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c3393ed28d242838d6f2f170184f7e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 9948d4a0552a4dfcad497d4fe1472fb1) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.433059 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4968ce56c6b4457097880a8ea7f924d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.436024 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1213) took 2428 ms. Trace:
I20250115 14:52:16.436609 1751 rpcz_store.cc:276] 0115 14:52:14.007799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.007852 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:16.435896 (+2428044us) ts_tablet_manager.cc:1145] Deleting tablet f2993e1c95234a1db2ae98a8e08c43f8
0115 14:52:16.435907 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.436007 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2427861}
W20250115 14:52:16.437263 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet f2993e1c95234a1db2ae98a8e08c43f8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2993e1c95234a1db2ae98a8e08c43f8 already in progress: creating tablet
I20250115 14:52:16.437651 1751 tablet_replica.cc:331] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.438237 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7db35ff0383549de86230bb1eb89389f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.438855 1751 raft_consensus.cc:2238] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.439141 1751 raft_consensus.cc:2267] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.439709 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_93b0e0aeb9414c3daff5c72ffe77258e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:16.439869 1751 ts_tablet_manager.cc:1905] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.439915 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_ebfbdc698c8c448ca2e60757b1fd574c [id=471b029f1dba4dd087deb339bba02013]) successfully deleted
W20250115 14:52:16.440106 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b3daf6416ad847aeaa83087cc81c9239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3daf6416ad847aeaa83087cc81c9239 already in progress: creating tablet
W20250115 14:52:16.440268 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93b0e0aeb9414c3daff5c72ffe77258e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.441462 1167 ts_tablet_manager.cc:1939] T 1140016d11a74a2cadba494c0b319c54 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:16.441865 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_51bdcc69abfc456cb288efd2b3db4a3b" } new_table_name: "default.Table_339586e5e5d246a88cad1293194ab296" modify_external_catalogs: true
I20250115 14:52:16.442180 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1246) took 1774 ms. Trace:
I20250115 14:52:16.442251 1167 rpcz_store.cc:276] 0115 14:52:14.667250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.870004 (+202754us) service_pool.cc:225] Handling call
0115 14:52:16.233105 (+1363101us) ts_tablet_manager.cc:1145] Deleting tablet 1140016d11a74a2cadba494c0b319c54
0115 14:52:16.233118 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.249000 (+ 15882us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.252019 (+ 3019us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.442160 (+190141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1362936}
I20250115 14:52:16.442567 1167 tablet_replica.cc:331] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:16.442734 1167 raft_consensus.cc:2238] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.442847 1167 raft_consensus.cc:2267] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.443287 1167 ts_tablet_manager.cc:1905] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.444444 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 1140016d11a74a2cadba494c0b319c54 (table default.Table_06fdf72400894754b2da4a05ae517fac [id=fa2c05b8018545958699fafc77f6e832]) successfully deleted
I20250115 14:52:16.444793 2774 tablet.cc:1721] T e0eba208f9be4a588cc81e4d99336eac P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:16.449930 1751 ts_tablet_manager.cc:1918] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.453457 2849 tablet.cc:1721] T e0eba208f9be4a588cc81e4d99336eac P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:16.454581 1751 log.cc:1198] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4adf1353bd554d8294a7f570931095d2
I20250115 14:52:16.455035 1751 ts_tablet_manager.cc:1939] T 4adf1353bd554d8294a7f570931095d2 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.455749 1168 tablet_bootstrap.cc:492] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:16.455870 1168 ts_tablet_manager.cc:1397] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.002s
I20250115 14:52:16.456532 1168 raft_consensus.cc:357] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.456719 1168 raft_consensus.cc:383] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.456763 1168 raft_consensus.cc:738] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.457465 2872 tablet.cc:1721] T e0eba208f9be4a588cc81e4d99336eac P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250115 14:52:16.457721 1168 consensus_queue.cc:260] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.466204 2865 raft_consensus.cc:491] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.466351 2865 raft_consensus.cc:513] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.467499 2865 leader_election.cc:290] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.468602 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8da9bd5ec841ebbc61b0f8de5ace3b" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:16.468895 2872 tablet.cc:1721] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:16.468384 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8da9bd5ec841ebbc61b0f8de5ace3b" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:16.471735 710 leader_election.cc:304] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:16.471969 2865 raft_consensus.cc:2743] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.472949 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1214) took 2414 ms. Trace:
I20250115 14:52:16.475039 1751 rpcz_store.cc:276] 0115 14:52:14.058294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.058346 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:16.437599 (+2379253us) ts_tablet_manager.cc:1145] Deleting tablet 4adf1353bd554d8294a7f570931095d2
0115 14:52:16.437610 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.447095 (+ 9485us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.449862 (+ 2767us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.472928 (+ 23066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2379066}
W20250115 14:52:16.440268 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7c3393ed28d242838d6f2f170184f7e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c3393ed28d242838d6f2f170184f7e8 already in progress: creating tablet
W20250115 14:52:16.475806 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4968ce56c6b4457097880a8ea7f924d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4968ce56c6b4457097880a8ea7f924d8 already in progress: creating tablet
I20250115 14:52:16.475960 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 4adf1353bd554d8294a7f570931095d2 (table default.Table_0e5d68124c7a4a0dbde272f4fb012cea [id=1c9959ef09ed4988baf48112f0280303]) successfully deleted
I20250115 14:52:16.476388 1168 ts_tablet_manager.cc:1428] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250115 14:52:16.477257 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1216) took 2208 ms. Trace:
I20250115 14:52:16.477365 1751 rpcz_store.cc:276] 0115 14:52:14.267961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.268024 (+ 63us) service_pool.cc:225] Handling call
0115 14:52:16.476326 (+2208302us) ts_tablet_manager.cc:1145] Deleting tablet bfc6e5c81a6d40a3a2054450d0478ad6
0115 14:52:16.476336 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.476432 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2208121}
W20250115 14:52:16.477743 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet bfc6e5c81a6d40a3a2054450d0478ad6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfc6e5c81a6d40a3a2054450d0478ad6 already in progress: creating tablet
I20250115 14:52:16.478025 1751 tablet_replica.cc:331] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.478384 1751 raft_consensus.cc:2238] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.478539 1751 raft_consensus.cc:2267] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.479065 1751 ts_tablet_manager.cc:1905] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:16.475978 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7db35ff0383549de86230bb1eb89389f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db35ff0383549de86230bb1eb89389f already in progress: creating tablet
W20250115 14:52:16.479677 441 consensus_peers.cc:487] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b -> Peer 1cc2e23c560f4096be7b14006bdfc525 (127.0.104.60:40227): Couldn't send request to peer 1cc2e23c560f4096be7b14006bdfc525. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250115 14:52:16.481603 453 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ff284a245a940fcb4567f95cf8fd424 consensus state has no leader
I20250115 14:52:16.482003 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_75c1d7f0fd574f53a5b1a2853de03c30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.482545 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75c1d7f0fd574f53a5b1a2853de03c30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.483780 1168 tablet_bootstrap.cc:492] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:16.485256 1168 tablet_bootstrap.cc:654] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.490924 1751 ts_tablet_manager.cc:1918] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.494486 1751 log.cc:1198] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f3280e8ba6db4446aa3e3bfe2836abfa
I20250115 14:52:16.495193 1751 ts_tablet_manager.cc:1939] T f3280e8ba6db4446aa3e3bfe2836abfa P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.495990 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_73c030f6c4c44f9bb54b65a8356b9c87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:16.496807 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1215) took 2229 ms. Trace:
W20250115 14:52:16.497228 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73c030f6c4c44f9bb54b65a8356b9c87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.497265 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1033f77f38414857a1255c05f93935d2" } modify_external_catalogs: true
I20250115 14:52:16.497385 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1033f77f38414857a1255c05f93935d2" } modify_external_catalogs: true
W20250115 14:52:16.498558 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ca883d907214c27bfc119ef9dd687d5 has no consensus state
I20250115 14:52:16.498891 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbeda353ef0a4bc987d588b99ee57db2" } modify_external_catalogs: true
I20250115 14:52:16.499027 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_bbeda353ef0a4bc987d588b99ee57db2" } modify_external_catalogs: true
I20250115 14:52:16.500599 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_928cab0a21204c4d82a7bca5e4a0c193"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.501091 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_928cab0a21204c4d82a7bca5e4a0c193 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.501385 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 7d8da9bd5ec841ebbc61b0f8de5ace3b (table default.Table_f634603ea42b4215b8ef3df8c446ada7 [id=174b82e1e06f4073a74a25aad8fc8045]). Expected version 2 got 0
W20250115 14:52:16.502564 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4536bb3d44b40d2825e1f4e59d8a148 consensus state has no leader
I20250115 14:52:16.503826 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 325cd3fd8b4d447998822163e575e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.505781 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1140016d11a74a2cadba494c0b319c54 with delete_type TABLET_DATA_DELETED (replaced by tablet 068771fee92f403d8997646291a83dc6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.505995 1166 tablet_replica.cc:331] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.506175 1166 raft_consensus.cc:2238] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.506330 1166 raft_consensus.cc:2267] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.507437 1166 ts_tablet_manager.cc:1905] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.509354 2549 raft_consensus.cc:513] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.510234 2549 leader_election.cc:290] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.510670 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet bf648bb656634def8189fb514a07e144 (table default.Table_fd84fd3d47974e05a580fae352838c20 [id=a00115ecf6954f83b3118654f98e629d]). Expected version 1 got 0
I20250115 14:52:16.510891 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 246720dbb4ed4216ac87fb4a3df79e2c (table default.Table_daaa2755fe9d4f54ae6ff33ae452ec05 [id=8c0229e6f6884ea7be3417678bb27377]). Expected version 1 got 0
I20250115 14:52:16.511246 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c04fa23e7c40a5b5e07852e5d74a27" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:16.511754 848 leader_election.cc:343] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.519884 1166 ts_tablet_manager.cc:1918] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.519989 1166 log.cc:1198] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/325cd3fd8b4d447998822163e575e758
I20250115 14:52:16.520223 1164 tablet_bootstrap.cc:492] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:16.520258 1166 ts_tablet_manager.cc:1939] T 325cd3fd8b4d447998822163e575e758 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:16.520360 1164 ts_tablet_manager.cc:1397] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.147s user 0.025s sys 0.025s
I20250115 14:52:16.520968 1164 raft_consensus.cc:357] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.521159 1164 raft_consensus.cc:383] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.521222 1164 raft_consensus.cc:738] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.521416 1164 consensus_queue.cc:260] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.521230 1166 tablet_replica.cc:331] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.521883 1164 ts_tablet_manager.cc:1428] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:16.522140 1164 tablet_bootstrap.cc:492] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:16.522351 1166 raft_consensus.cc:2238] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.522464 1166 raft_consensus.cc:2267] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.522964 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet d27a88650b024af19938618d9ed0cc23
I20250115 14:52:16.523157 1166 ts_tablet_manager.cc:1905] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.524278 890 tablet_service.cc:1514] Processing DeleteTablet for tablet d27a88650b024af19938618d9ed0cc23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.524991 742 tablet_service.cc:1514] Processing DeleteTablet for tablet d27a88650b024af19938618d9ed0cc23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.525014 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 325cd3fd8b4d447998822163e575e758 (table default.Table_52bf98f0d5b44013ba19e15f0fc24cce [id=c7768272a8d14ac0aba3f569f8c8f58e]) successfully deleted
I20250115 14:52:16.527025 2865 raft_consensus.cc:491] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.527143 2865 raft_consensus.cc:513] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.527674 2865 leader_election.cc:290] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.527866 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet afea664025e54f6099d825c8f7a0ea1a
I20250115 14:52:16.528646 890 tablet_service.cc:1514] Processing DeleteTablet for tablet afea664025e54f6099d825c8f7a0ea1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:16.528900 710 leader_election.cc:343] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 248cadcdef8b4e99923cdbca39690685
W20250115 14:52:16.529078 710 leader_election.cc:343] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 248cadcdef8b4e99923cdbca39690685
I20250115 14:52:16.529155 710 leader_election.cc:304] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:16.529600 742 tablet_service.cc:1514] Processing DeleteTablet for tablet afea664025e54f6099d825c8f7a0ea1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.530206 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_f634603ea42b4215b8ef3df8c446ada7 [id=174b82e1e06f4073a74a25aad8fc8045] alter complete (version 2)
I20250115 14:52:16.530467 1167 ts_tablet_manager.cc:1918] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.530568 1167 log.cc:1198] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/325cd3fd8b4d447998822163e575e758
I20250115 14:52:16.530649 2865 raft_consensus.cc:2743] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.530823 1167 ts_tablet_manager.cc:1939] T 325cd3fd8b4d447998822163e575e758 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:16.532166 453 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_b578368ee37e47b6a6dc45444cf2ee8e [id=6d0e848982e245e1870c661f7dc35138] alter complete (version 3)
I20250115 14:52:16.532315 1168 tablet_bootstrap.cc:492] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:16.532428 1168 ts_tablet_manager.cc:1397] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20250115 14:52:16.497560 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet e910c387e6a24caf95e14b57c8b83e6f with delete_type TABLET_DATA_DELETED (replaced by tablet bb859e2a553f4ab3b20452b48e933f0e) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.510771 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86c04fa23e7c40a5b5e07852e5d74a27" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:16.532860 1046 raft_consensus.cc:3054] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.533108 1168 raft_consensus.cc:357] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.533267 1168 raft_consensus.cc:383] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.533308 1168 raft_consensus.cc:738] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.533484 1168 consensus_queue.cc:260] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.533619 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 9494c0c57be8499a883888d067c7db77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.533932 1168 ts_tablet_manager.cc:1428] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250115 14:52:16.534266 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9ce226d3ba44d6908257bbd0df3a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.534468 2831 raft_consensus.cc:491] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.534526 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1248) took 1867 ms. Trace:
I20250115 14:52:16.534595 1167 rpcz_store.cc:276] 0115 14:52:14.667368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.870181 (+202813us) service_pool.cc:225] Handling call
0115 14:52:16.442526 (+1572345us) ts_tablet_manager.cc:1145] Deleting tablet 325cd3fd8b4d447998822163e575e758
0115 14:52:16.442534 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.494823 (+ 52289us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.530368 (+ 35545us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.534510 (+ 4142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1572235}
I20250115 14:52:16.534570 2831 raft_consensus.cc:513] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.535187 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1249) took 1867 ms. Trace:
I20250115 14:52:16.535251 1167 rpcz_store.cc:276] 0115 14:52:14.667425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.870302 (+202877us) service_pool.cc:225] Handling call
0115 14:52:16.535075 (+1664773us) ts_tablet_manager.cc:1145] Deleting tablet c9a5593068d54d7bb3874cf434505e73
0115 14:52:16.535084 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.535171 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1664660}
I20250115 14:52:16.535327 2831 leader_election.cc:290] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.535529 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 325cd3fd8b4d447998822163e575e758 (table default.Table_52bf98f0d5b44013ba19e15f0fc24cce [id=c7768272a8d14ac0aba3f569f8c8f58e]) successfully deleted
W20250115 14:52:16.535768 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c9a5593068d54d7bb3874cf434505e73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a5593068d54d7bb3874cf434505e73 already in progress: creating tablet
I20250115 14:52:16.535791 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9ce226d3ba44d6908257bbd0df3a12" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:16.536219 983 leader_election.cc:343] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.535346 1046 raft_consensus.cc:2463] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:16.536612 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9ce226d3ba44d6908257bbd0df3a12" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.537016 847 leader_election.cc:304] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:16.537091 984 leader_election.cc:343] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.537191 984 leader_election.cc:304] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.522203 1162 tablet_bootstrap.cc:492] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.524323 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet d27a88650b024af19938618d9ed0cc23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.528771 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet afea664025e54f6099d825c8f7a0ea1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.537642 2831 raft_consensus.cc:2743] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.537698 1162 ts_tablet_manager.cc:1397] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.136s user 0.011s sys 0.017s
I20250115 14:52:16.537715 2835 raft_consensus.cc:2798] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:16.537817 2835 raft_consensus.cc:695] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:16.538028 2835 consensus_queue.cc:237] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.538512 1162 raft_consensus.cc:357] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.538758 1162 raft_consensus.cc:383] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.538865 1162 raft_consensus.cc:738] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.539141 1162 consensus_queue.cc:260] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.540376 1162 ts_tablet_manager.cc:1428] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:16.540503 1751 rpcz_store.cc:276] 0115 14:52:14.267830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.270667 (+ 2837us) service_pool.cc:225] Handling call
0115 14:52:16.477981 (+2207314us) ts_tablet_manager.cc:1145] Deleting tablet f3280e8ba6db4446aa3e3bfe2836abfa
0115 14:52:16.477990 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.487908 (+ 9918us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.490813 (+ 2905us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.496788 (+ 5975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2207132}
I20250115 14:52:16.540993 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f3280e8ba6db4446aa3e3bfe2836abfa (table default.Table_5da68f9f7cdb439daae3b29bc1e40438 [id=3e382235d0e34eb6aad49f9c52df8603]) successfully deleted
I20250115 14:52:16.541396 1162 tablet_bootstrap.cc:492] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.544046 1751 tablet_replica.cc:331] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.544308 1751 raft_consensus.cc:2238] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.545811 1751 raft_consensus.cc:2267] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.546420 1751 ts_tablet_manager.cc:1905] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.546777 1162 tablet_bootstrap.cc:654] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.547650 1168 tablet_bootstrap.cc:492] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:16.549021 1168 tablet_bootstrap.cc:654] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.552170 1166 ts_tablet_manager.cc:1918] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.553848 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7f957146690846bd9bf51f27431bd54d" } modify_external_catalogs: true
I20250115 14:52:16.554035 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7f957146690846bd9bf51f27431bd54d" } modify_external_catalogs: true
I20250115 14:52:16.554665 1166 log.cc:1198] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/1140016d11a74a2cadba494c0b319c54
I20250115 14:52:16.554984 1166 ts_tablet_manager.cc:1939] T 1140016d11a74a2cadba494c0b319c54 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:16.562125 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 1140016d11a74a2cadba494c0b319c54 (table default.Table_06fdf72400894754b2da4a05ae517fac [id=fa2c05b8018545958699fafc77f6e832]) successfully deleted
W20250115 14:52:16.562394 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet d27a88650b024af19938618d9ed0cc23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d27a88650b024af19938618d9ed0cc23 already in progress: creating tablet
W20250115 14:52:16.562610 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet afea664025e54f6099d825c8f7a0ea1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afea664025e54f6099d825c8f7a0ea1a already in progress: creating tablet
I20250115 14:52:16.563299 1162 tablet_bootstrap.cc:492] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.563474 1162 ts_tablet_manager.cc:1397] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250115 14:52:16.564064 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ffa9cd1279fe4e0c9168472993a983dd
I20250115 14:52:16.564224 1162 raft_consensus.cc:357] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.564412 1162 raft_consensus.cc:383] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.564465 1162 raft_consensus.cc:738] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.564677 1162 consensus_queue.cc:260] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.565241 1162 ts_tablet_manager.cc:1428] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:16.565474 1162 tablet_bootstrap.cc:492] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.566344 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ffa9cd1279fe4e0c9168472993a983dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.566478 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet ffa9cd1279fe4e0c9168472993a983dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.566990 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ffa9cd1279fe4e0c9168472993a983dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.568354 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 0cae4c9228c640e495507e8f7ad9f544 from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:16.568434 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ffa9cd1279fe4e0c9168472993a983dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffa9cd1279fe4e0c9168472993a983dd already in progress: creating tablet
I20250115 14:52:16.569842 1751 ts_tablet_manager.cc:1918] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.569952 1751 log.cc:1198] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/36de948083a946eba1a96f276e4f4f91
I20250115 14:52:16.570226 1751 ts_tablet_manager.cc:1939] T 36de948083a946eba1a96f276e4f4f91 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.570940 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1217) took 2288 ms. Trace:
I20250115 14:52:16.571022 1751 rpcz_store.cc:276] 0115 14:52:14.282608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.282660 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:16.543995 (+2261335us) ts_tablet_manager.cc:1145] Deleting tablet 36de948083a946eba1a96f276e4f4f91
0115 14:52:16.544005 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.558420 (+ 14415us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.569729 (+ 11309us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.570923 (+ 1194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2261135}
I20250115 14:52:16.571300 1162 tablet_bootstrap.cc:654] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.571483 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 36de948083a946eba1a96f276e4f4f91 (table default.Table_89373c553f9c407a9e205bfe0d1acf99 [id=26c53728ab22430b95ca322543118a43]) successfully deleted
I20250115 14:52:16.571811 1751 tablet_replica.cc:331] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.572047 1751 raft_consensus.cc:2238] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.572259 1751 raft_consensus.cc:2267] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.572849 1751 ts_tablet_manager.cc:1905] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.573105 1164 tablet_bootstrap.cc:654] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.574420 1168 tablet_bootstrap.cc:492] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:16.574559 1168 ts_tablet_manager.cc:1397] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.000s
I20250115 14:52:16.575245 1168 raft_consensus.cc:357] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.575429 1168 raft_consensus.cc:383] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.575474 1168 raft_consensus.cc:738] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.575702 1168 consensus_queue.cc:260] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.579151 1751 ts_tablet_manager.cc:1918] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.579227 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1254) took 1753 ms. Trace:
I20250115 14:52:16.579330 1751 log.cc:1198] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d1259256b7744e739581125611abbe67
I20250115 14:52:16.579306 1167 rpcz_store.cc:276] 0115 14:52:14.825804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.880611 (+ 54807us) service_pool.cc:225] Handling call
0115 14:52:16.579099 (+1698488us) ts_tablet_manager.cc:1145] Deleting tablet 5b33c876fab1476aba3b1f6e799fa96c
0115 14:52:16.579109 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.579209 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1698265}
W20250115 14:52:16.579702 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 5b33c876fab1476aba3b1f6e799fa96c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b33c876fab1476aba3b1f6e799fa96c already in progress: creating tablet
I20250115 14:52:16.579756 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1255) took 1753 ms. Trace:
I20250115 14:52:16.579813 1167 rpcz_store.cc:276] 0115 14:52:14.825894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.880852 (+ 54958us) service_pool.cc:225] Handling call
0115 14:52:16.579658 (+1698806us) ts_tablet_manager.cc:1145] Deleting tablet db57f7c3a4c14cc19d3a0624e5b1817a
0115 14:52:16.579666 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.579742 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1698627}
W20250115 14:52:16.580121 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet db57f7c3a4c14cc19d3a0624e5b1817a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db57f7c3a4c14cc19d3a0624e5b1817a already in progress: creating tablet
I20250115 14:52:16.580161 1167 tablet_replica.cc:331] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:16.580540 1751 ts_tablet_manager.cc:1939] T d1259256b7744e739581125611abbe67 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.581461 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1218) took 2240 ms. Trace:
I20250115 14:52:16.582245 1751 rpcz_store.cc:276] 0115 14:52:14.341269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.341323 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:16.571763 (+2230440us) ts_tablet_manager.cc:1145] Deleting tablet d1259256b7744e739581125611abbe67
0115 14:52:16.571773 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.576112 (+ 4339us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.579065 (+ 2953us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.581443 (+ 2378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230244}
I20250115 14:52:16.582751 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d1259256b7744e739581125611abbe67 (table default.Table_28a52958055546da964198b7c17f8b8c [id=d485b796d6b349d9b6de80eec41a36c7]) successfully deleted
I20250115 14:52:16.583122 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1223) took 2094 ms. Trace:
I20250115 14:52:16.583235 1751 rpcz_store.cc:276] 0115 14:52:14.488384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.492390 (+ 4006us) service_pool.cc:225] Handling call
0115 14:52:16.583013 (+2090623us) ts_tablet_manager.cc:1145] Deleting tablet 7830c683a109448ab06c15b5de692e8b
0115 14:52:16.583022 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.583108 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2090452}
W20250115 14:52:16.583631 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7830c683a109448ab06c15b5de692e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7830c683a109448ab06c15b5de692e8b already in progress: creating tablet
I20250115 14:52:16.582192 1162 tablet_bootstrap.cc:492] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.583935 1162 ts_tablet_manager.cc:1397] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.001s
I20250115 14:52:16.584770 1162 raft_consensus.cc:357] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.585227 1162 raft_consensus.cc:383] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.585318 1162 raft_consensus.cc:738] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.585148 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1224) took 2096 ms. Trace:
I20250115 14:52:16.585522 1751 rpcz_store.cc:276] 0115 14:52:14.488502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.492600 (+ 4098us) service_pool.cc:225] Handling call
0115 14:52:16.585053 (+2092453us) ts_tablet_manager.cc:1145] Deleting tablet 061234cb76b842a7a374ddd5f44fd9a1
0115 14:52:16.585062 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.585135 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2092347}
W20250115 14:52:16.585923 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 061234cb76b842a7a374ddd5f44fd9a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 061234cb76b842a7a374ddd5f44fd9a1 already in progress: creating tablet
I20250115 14:52:16.586220 1751 tablet_replica.cc:331] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.586418 1751 raft_consensus.cc:2238] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.586773 1162 consensus_queue.cc:260] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.587038 1751 raft_consensus.cc:2267] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.587723 1751 ts_tablet_manager.cc:1905] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.594166 1751 ts_tablet_manager.cc:1918] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.588106 1162 ts_tablet_manager.cc:1428] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250115 14:52:16.594571 1162 tablet_bootstrap.cc:492] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.595182 1167 raft_consensus.cc:2238] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.595371 1167 raft_consensus.cc:2267] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.595964 1167 ts_tablet_manager.cc:1905] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.596573 1162 tablet_bootstrap.cc:654] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.597787 1751 log.cc:1198] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d6f2bc62621d402cacfcbe2eb74173f9
I20250115 14:52:16.598193 1751 ts_tablet_manager.cc:1939] T d6f2bc62621d402cacfcbe2eb74173f9 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.599186 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1225) took 2110 ms. Trace:
I20250115 14:52:16.600047 1751 rpcz_store.cc:276] 0115 14:52:14.488604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.492730 (+ 4126us) service_pool.cc:225] Handling call
0115 14:52:16.586180 (+2093450us) ts_tablet_manager.cc:1145] Deleting tablet d6f2bc62621d402cacfcbe2eb74173f9
0115 14:52:16.586189 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.591236 (+ 5047us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.594090 (+ 2854us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.599168 (+ 5078us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":122,"tablet-delete.queue_time_us":2093337}
I20250115 14:52:16.600754 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d6f2bc62621d402cacfcbe2eb74173f9 (table default.Table_58c17e0ad22e41738519c725c25dcdfc [id=3249c65f52624113be96f7ef7b828854]) successfully deleted
I20250115 14:52:16.603245 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1227) took 2102 ms. Trace:
I20250115 14:52:16.603611 1751 rpcz_store.cc:276] 0115 14:52:14.500944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.501106 (+ 162us) service_pool.cc:225] Handling call
0115 14:52:16.603130 (+2102024us) ts_tablet_manager.cc:1145] Deleting tablet 13a4837b72eb4008ad612a9fc86dfac7
0115 14:52:16.603140 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.603229 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2101857}
W20250115 14:52:16.604236 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 13a4837b72eb4008ad612a9fc86dfac7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13a4837b72eb4008ad612a9fc86dfac7 already in progress: creating tablet
I20250115 14:52:16.604768 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1226) took 2104 ms. Trace:
I20250115 14:52:16.604877 1751 rpcz_store.cc:276] 0115 14:52:14.500818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.501305 (+ 487us) service_pool.cc:225] Handling call
0115 14:52:16.604672 (+2103367us) ts_tablet_manager.cc:1145] Deleting tablet 9ad3cda1056349a789a1751c5d386d0c
0115 14:52:16.604678 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.604752 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2103261}
W20250115 14:52:16.605355 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 9ad3cda1056349a789a1751c5d386d0c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ad3cda1056349a789a1751c5d386d0c already in progress: creating tablet
I20250115 14:52:16.605703 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1228) took 2104 ms. Trace:
I20250115 14:52:16.605799 1751 rpcz_store.cc:276] 0115 14:52:14.501056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.501427 (+ 371us) service_pool.cc:225] Handling call
0115 14:52:16.605615 (+2104188us) ts_tablet_manager.cc:1145] Deleting tablet 24729156769443f8926577bae7707847
0115 14:52:16.605624 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.605689 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2104100}
W20250115 14:52:16.606197 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 24729156769443f8926577bae7707847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24729156769443f8926577bae7707847 already in progress: creating tablet
I20250115 14:52:16.606493 1751 tablet_replica.cc:331] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.606729 1751 raft_consensus.cc:2238] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.607039 1751 raft_consensus.cc:2267] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.607347 2831 raft_consensus.cc:491] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.607470 2831 raft_consensus.cc:513] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.607599 1751 ts_tablet_manager.cc:1905] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.608031 2831 leader_election.cc:290] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.608633 1168 ts_tablet_manager.cc:1428] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20250115 14:52:16.608989 1168 tablet_bootstrap.cc:492] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:16.610388 1168 tablet_bootstrap.cc:654] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.610297 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1df21ab18f45a8adf9fedd22b2b8a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.608646 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1df21ab18f45a8adf9fedd22b2b8a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:16.610836 911 raft_consensus.cc:2463] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:16.611281 983 leader_election.cc:304] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:52:16.612134 984 leader_election.cc:343] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.612202 1162 tablet_bootstrap.cc:492] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.612362 1162 ts_tablet_manager.cc:1397] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20250115 14:52:16.612427 1164 tablet_bootstrap.cc:492] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:16.612552 1164 ts_tablet_manager.cc:1397] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.000s
I20250115 14:52:16.613011 1162 raft_consensus.cc:357] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.613176 1162 raft_consensus.cc:383] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.613222 1162 raft_consensus.cc:738] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.613176 1164 raft_consensus.cc:357] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.613334 1164 raft_consensus.cc:383] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.613384 1164 raft_consensus.cc:738] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.613466 1751 ts_tablet_manager.cc:1918] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:16.613535 1751 log.cc:1198] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/1298fdd0217e4df6bfbafc49b2f2c65c
I20250115 14:52:16.613430 1162 consensus_queue.cc:260] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.613564 1164 consensus_queue.cc:260] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:16.613682 2839 raft_consensus.cc:2798] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:16.613736 2839 raft_consensus.cc:491] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:16.613790 2839 raft_consensus.cc:3054] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.613843 1751 ts_tablet_manager.cc:1939] T 1298fdd0217e4df6bfbafc49b2f2c65c P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.614316 1164 ts_tablet_manager.cc:1428] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:16.614567 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1251) took 1942 ms. Trace:
I20250115 14:52:16.614635 1751 rpcz_store.cc:276] 0115 14:52:14.672387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.672432 (+ 45us) service_pool.cc:225] Handling call
0115 14:52:16.606452 (+1934020us) ts_tablet_manager.cc:1145] Deleting tablet 1298fdd0217e4df6bfbafc49b2f2c65c
0115 14:52:16.606461 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.610655 (+ 4194us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.613385 (+ 2730us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.614550 (+ 1165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1933854}
I20250115 14:52:16.615063 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 1298fdd0217e4df6bfbafc49b2f2c65c (table default.Table_15773744107143d6b1e5215c832f0a3f [id=f4305d1bfe12463fa68fb1405be5267f]) successfully deleted
I20250115 14:52:16.615439 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1252) took 1942 ms. Trace:
I20250115 14:52:16.615525 1751 rpcz_store.cc:276] 0115 14:52:14.672710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.672746 (+ 36us) service_pool.cc:225] Handling call
0115 14:52:16.615336 (+1942590us) ts_tablet_manager.cc:1145] Deleting tablet c9a5593068d54d7bb3874cf434505e73
0115 14:52:16.615344 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.615424 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942471}
W20250115 14:52:16.615932 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c9a5593068d54d7bb3874cf434505e73 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a5593068d54d7bb3874cf434505e73 already in progress: creating tablet
I20250115 14:52:16.616132 1164 tablet_bootstrap.cc:492] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:16.616252 1751 tablet_replica.cc:331] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.616453 1751 raft_consensus.cc:2238] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.616603 1751 raft_consensus.cc:2267] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.617218 1751 ts_tablet_manager.cc:1905] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.616137 2839 raft_consensus.cc:513] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.617821 2839 leader_election.cc:290] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.617970 1168 tablet_bootstrap.cc:492] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:16.618105 1168 ts_tablet_manager.cc:1397] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250115 14:52:16.618383 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1df21ab18f45a8adf9fedd22b2b8a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:16.618556 911 raft_consensus.cc:3054] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.618621 1162 ts_tablet_manager.cc:1428] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250115 14:52:16.619513 2839 raft_consensus.cc:491] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.619623 2839 raft_consensus.cc:513] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.620573 2839 leader_election.cc:290] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.620554 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400880835ab4efdb2af0be5553d20dc" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:16.620683 1162 tablet_bootstrap.cc:492] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.620990 911 raft_consensus.cc:2463] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
W20250115 14:52:16.621065 983 leader_election.cc:343] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.621784 983 leader_election.cc:304] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.622004 1162 tablet_bootstrap.cc:654] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.622012 1164 tablet_bootstrap.cc:654] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.622835 1168 raft_consensus.cc:357] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.623008 1168 raft_consensus.cc:383] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.623063 1168 raft_consensus.cc:738] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.623333 1751 ts_tablet_manager.cc:1918] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.623255 1168 consensus_queue.cc:260] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.623493 1751 log.cc:1198] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ba33eb3f5a40450db5cf115fbe8a17e7
I20250115 14:52:16.623775 1751 ts_tablet_manager.cc:1939] T ba33eb3f5a40450db5cf115fbe8a17e7 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.623756 1168 ts_tablet_manager.cc:1428] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250115 14:52:16.624593 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1255) took 1950 ms. Trace:
I20250115 14:52:16.624712 1751 rpcz_store.cc:276] 0115 14:52:14.673968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.674011 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:16.616208 (+1942197us) ts_tablet_manager.cc:1145] Deleting tablet ba33eb3f5a40450db5cf115fbe8a17e7
0115 14:52:16.616217 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.620037 (+ 3820us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.623241 (+ 3204us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.624576 (+ 1335us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1942030}
I20250115 14:52:16.625231 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ba33eb3f5a40450db5cf115fbe8a17e7 (table default.Table_ff33645127a443efb2b776d808942dba [id=f8c4ca50220a4dbf9b9480213d836e5f]) successfully deleted
I20250115 14:52:16.625707 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1250) took 1953 ms. Trace:
I20250115 14:52:16.625828 1751 rpcz_store.cc:276] 0115 14:52:14.672279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.695437 (+ 23158us) service_pool.cc:225] Handling call
0115 14:52:16.625543 (+1930106us) ts_tablet_manager.cc:1145] Deleting tablet abefc717e0eb41b18725ef0c73154fdd
0115 14:52:16.625553 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.625689 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1929852}
W20250115 14:52:16.626323 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet abefc717e0eb41b18725ef0c73154fdd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abefc717e0eb41b18725ef0c73154fdd already in progress: creating tablet
I20250115 14:52:16.626782 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1249) took 1954 ms. Trace:
I20250115 14:52:16.626890 1751 rpcz_store.cc:276] 0115 14:52:14.672182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.725310 (+ 53128us) service_pool.cc:225] Handling call
0115 14:52:16.626689 (+1901379us) ts_tablet_manager.cc:1145] Deleting tablet 3e834623f9c7482d965afd23c89a2510
0115 14:52:16.626698 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.626767 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1901143}
W20250115 14:52:16.627272 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 3e834623f9c7482d965afd23c89a2510 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e834623f9c7482d965afd23c89a2510 already in progress: creating tablet
I20250115 14:52:16.627686 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1259) took 1896 ms. Trace:
I20250115 14:52:16.627777 1751 rpcz_store.cc:276] 0115 14:52:14.731466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.731509 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:16.627599 (+1896090us) ts_tablet_manager.cc:1145] Deleting tablet db57f7c3a4c14cc19d3a0624e5b1817a
0115 14:52:16.627608 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.627672 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1895912}
I20250115 14:52:16.628082 2839 raft_consensus.cc:2798] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
W20250115 14:52:16.628144 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet db57f7c3a4c14cc19d3a0624e5b1817a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db57f7c3a4c14cc19d3a0624e5b1817a already in progress: creating tablet
I20250115 14:52:16.628283 2839 raft_consensus.cc:695] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:16.628479 1751 tablet_replica.cc:331] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.622466 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f400880835ab4efdb2af0be5553d20dc" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.622466 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1df21ab18f45a8adf9fedd22b2b8a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:16.628803 1751 raft_consensus.cc:2238] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.628983 1751 raft_consensus.cc:2267] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:16.629135 984 leader_election.cc:343] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:16.629456 984 leader_election.cc:343] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.629556 984 leader_election.cc:304] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.629709 1751 ts_tablet_manager.cc:1905] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.629910 2831 raft_consensus.cc:2743] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.628551 2839 consensus_queue.cc:237] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.632128 1162 tablet_bootstrap.cc:492] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.632318 1162 ts_tablet_manager.cc:1397] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.002s
I20250115 14:52:16.632531 1164 tablet_bootstrap.cc:492] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:16.632663 1164 ts_tablet_manager.cc:1397] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250115 14:52:16.633090 1162 raft_consensus.cc:357] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.633303 1162 raft_consensus.cc:383] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.634061 1162 raft_consensus.cc:738] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.634317 1162 consensus_queue.cc:260] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.634521 1168 tablet_bootstrap.cc:492] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:16.634855 1162 ts_tablet_manager.cc:1428] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:16.635135 1162 tablet_bootstrap.cc:492] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.636590 1162 tablet_bootstrap.cc:654] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.637813 1168 tablet_bootstrap.cc:654] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.633270 1164 raft_consensus.cc:357] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.639422 1164 raft_consensus.cc:383] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.639498 1164 raft_consensus.cc:738] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.640686 774 raft_consensus.cc:3054] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.642527 1751 ts_tablet_manager.cc:1918] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.642695 1751 log.cc:1198] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e6978223c1984a38bec5f2d06a9cf457
I20250115 14:52:16.643059 1751 ts_tablet_manager.cc:1939] T e6978223c1984a38bec5f2d06a9cf457 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.643462 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3caaaaf71eae487480e4b55631f7e099" } modify_external_catalogs: true
I20250115 14:52:16.643658 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3caaaaf71eae487480e4b55631f7e099" } modify_external_catalogs: true
I20250115 14:52:16.643743 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_def3b3bdfb824258849c1d6a35f228a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:16.643863 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1258) took 1912 ms. Trace:
I20250115 14:52:16.643942 1751 rpcz_store.cc:276] 0115 14:52:14.731377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.740148 (+ 8771us) service_pool.cc:225] Handling call
0115 14:52:16.628438 (+1888290us) ts_tablet_manager.cc:1145] Deleting tablet e6978223c1984a38bec5f2d06a9cf457
0115 14:52:16.628446 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.639134 (+ 10688us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.642245 (+ 3111us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.643842 (+ 1597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1888098}
W20250115 14:52:16.644258 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_def3b3bdfb824258849c1d6a35f228a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.644399 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e6978223c1984a38bec5f2d06a9cf457 (table default.Table_a1dc2d81847c4f31bc83b06081b8274f [id=e03ea31f74914ceba6e5a36ba1b30dc8]) successfully deleted
I20250115 14:52:16.644505 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_0c704df3a19a436e891ea2813ceb7db3" } new_table_name: "default.Table_46e48b62654f4b409c83d10831f0499d" modify_external_catalogs: true
I20250115 14:52:16.644805 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1257) took 1913 ms. Trace:
I20250115 14:52:16.644886 1751 rpcz_store.cc:276] 0115 14:52:14.731283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.745876 (+ 14593us) service_pool.cc:225] Handling call
0115 14:52:16.644671 (+1898795us) ts_tablet_manager.cc:1145] Deleting tablet 5b33c876fab1476aba3b1f6e799fa96c
0115 14:52:16.644683 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.644787 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1898523}
W20250115 14:52:16.645284 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 5b33c876fab1476aba3b1f6e799fa96c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b33c876fab1476aba3b1f6e799fa96c already in progress: creating tablet
I20250115 14:52:16.645634 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_312f79187f444b58988d8cda414ce44c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.646091 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_312f79187f444b58988d8cda414ce44c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.644188 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5f71454d37804130bd571f372234edc7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.646415 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f71454d37804130bd571f372234edc7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.646953 1751 tablet_replica.cc:331] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.647130 1751 raft_consensus.cc:2238] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.647296 1751 raft_consensus.cc:2267] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.647490 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1b5ca8c3048846588478dbfbebf1ff82" } modify_external_catalogs: true
I20250115 14:52:16.647632 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1b5ca8c3048846588478dbfbebf1ff82" } modify_external_catalogs: true
I20250115 14:52:16.648087 1751 ts_tablet_manager.cc:1905] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:16.653121 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96a5ce44bcb948f9abff0c20682e787f consensus state has no leader
I20250115 14:52:16.653535 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a2dac92e95de48bfb2af2bdd00b50c33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.653990 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2dac92e95de48bfb2af2bdd00b50c33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.655735 1751 ts_tablet_manager.cc:1918] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.655885 1751 log.cc:1198] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/2560d42c15c64b4c9c0cfe95706b4d9e
I20250115 14:52:16.656219 1751 ts_tablet_manager.cc:1939] T 2560d42c15c64b4c9c0cfe95706b4d9e P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.657166 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1260) took 1891 ms. Trace:
I20250115 14:52:16.657303 1751 rpcz_store.cc:276] 0115 14:52:14.765484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.765531 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:16.646911 (+1881380us) ts_tablet_manager.cc:1145] Deleting tablet 2560d42c15c64b4c9c0cfe95706b4d9e
0115 14:52:16.646920 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.651943 (+ 5023us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.655649 (+ 3706us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.657144 (+ 1495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1881215}
I20250115 14:52:16.657796 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 2560d42c15c64b4c9c0cfe95706b4d9e (table default.Table_339586e5e5d246a88cad1293194ab296 [id=92ab45b4bf154e789e3a9b33c77d38c4]) successfully deleted
I20250115 14:52:16.658649 1751 tablet_replica.cc:331] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.660652 1751 raft_consensus.cc:2238] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.660887 1751 raft_consensus.cc:2267] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.661434 1751 ts_tablet_manager.cc:1905] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.666911 2872 tablet.cc:1721] T 7d8da9bd5ec841ebbc61b0f8de5ace3b P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:16.667709 1751 ts_tablet_manager.cc:1918] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.667928 1751 log.cc:1198] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6d564ec5da4e4a3a9b903a4d9bbdb63a
I20250115 14:52:16.668397 1751 ts_tablet_manager.cc:1939] T 6d564ec5da4e4a3a9b903a4d9bbdb63a P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.669282 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1261) took 1766 ms. Trace:
I20250115 14:52:16.669426 1751 rpcz_store.cc:276] 0115 14:52:14.902362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.902678 (+ 316us) service_pool.cc:225] Handling call
0115 14:52:16.658586 (+1755908us) ts_tablet_manager.cc:1145] Deleting tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a
0115 14:52:16.658606 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.664610 (+ 6004us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.667620 (+ 3010us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.669263 (+ 1643us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1755690}
I20250115 14:52:16.669974 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6d564ec5da4e4a3a9b903a4d9bbdb63a (table default.Table_61bd6e7ab702444ba0919bfb8efb04f0 [id=b8934690b09642ff81dcd01b665b6565]) successfully deleted
I20250115 14:52:16.670449 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1262) took 1652 ms. Trace:
I20250115 14:52:16.670562 1751 rpcz_store.cc:276] 0115 14:52:15.018118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.018172 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:16.670311 (+1652139us) ts_tablet_manager.cc:1145] Deleting tablet 00844e77811b4cbfb7240176055e2966
0115 14:52:16.670322 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.670433 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651972}
W20250115 14:52:16.670990 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 00844e77811b4cbfb7240176055e2966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00844e77811b4cbfb7240176055e2966 already in progress: creating tablet
I20250115 14:52:16.671780 1164 consensus_queue.cc:260] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.672437 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1264) took 1643 ms. Trace:
I20250115 14:52:16.672448 1164 ts_tablet_manager.cc:1428] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250115 14:52:16.672567 1751 rpcz_store.cc:276] 0115 14:52:15.028601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.028670 (+ 69us) service_pool.cc:225] Handling call
0115 14:52:16.672331 (+1643661us) ts_tablet_manager.cc:1145] Deleting tablet 1020437ffddf49cc846fcd3311892b46
0115 14:52:16.672340 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.672420 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643469}
W20250115 14:52:16.673035 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 1020437ffddf49cc846fcd3311892b46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1020437ffddf49cc846fcd3311892b46 already in progress: creating tablet
I20250115 14:52:16.673117 1164 tablet_bootstrap.cc:492] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:16.673390 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1263) took 1644 ms. Trace:
I20250115 14:52:16.673491 1751 rpcz_store.cc:276] 0115 14:52:15.028461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.029831 (+ 1370us) service_pool.cc:225] Handling call
0115 14:52:16.673299 (+1643468us) ts_tablet_manager.cc:1145] Deleting tablet 2b26b9b1caf748aba7d36a04fbba2a80
0115 14:52:16.673307 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.673377 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643235}
W20250115 14:52:16.673878 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2b26b9b1caf748aba7d36a04fbba2a80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b26b9b1caf748aba7d36a04fbba2a80 already in progress: creating tablet
I20250115 14:52:16.674158 1751 tablet_replica.cc:331] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.674371 1751 raft_consensus.cc:2238] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.674566 1751 raft_consensus.cc:2267] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.675326 1751 ts_tablet_manager.cc:1905] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.675740 1164 tablet_bootstrap.cc:654] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.681360 1751 ts_tablet_manager.cc:1918] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:16.681514 1751 log.cc:1198] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/70425c25919b4d0b999450210e1cb33f
I20250115 14:52:16.681861 1751 ts_tablet_manager.cc:1939] T 70425c25919b4d0b999450210e1cb33f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.683225 1162 tablet_bootstrap.cc:492] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.683357 1162 ts_tablet_manager.cc:1397] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.017s
I20250115 14:52:16.683995 1162 raft_consensus.cc:357] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.684170 1162 raft_consensus.cc:383] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.684237 1162 raft_consensus.cc:738] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.684666 1162 consensus_queue.cc:260] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.685431 1162 ts_tablet_manager.cc:1428] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:16.685757 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f4536e6b631a4f40acf32c7fe4c42f10" } new_table_name: "default.Table_759d63a9392e477aa127e0d69c1c30ef" modify_external_catalogs: true
I20250115 14:52:16.686525 1164 tablet_bootstrap.cc:492] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:16.686641 1164 ts_tablet_manager.cc:1397] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250115 14:52:16.688114 1162 tablet_bootstrap.cc:492] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.689577 1162 tablet_bootstrap.cc:654] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.689754 1164 raft_consensus.cc:357] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:16.694038 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2087b4ce466f4167b7c32c7074dcf6e7 consensus state has no leader
I20250115 14:52:16.695219 1168 tablet_bootstrap.cc:492] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:16.695356 1168 ts_tablet_manager.cc:1397] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20250115 14:52:16.696092 1168 raft_consensus.cc:357] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.696303 1168 raft_consensus.cc:383] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.696364 1168 raft_consensus.cc:738] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.696485 1164 raft_consensus.cc:383] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.696600 1168 consensus_queue.cc:260] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.696812 1164 raft_consensus.cc:738] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.697088 1168 ts_tablet_manager.cc:1428] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:16.697353 1164 consensus_queue.cc:260] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.697784 1168 tablet_bootstrap.cc:492] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:16.698148 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1266) took 1663 ms. Trace:
I20250115 14:52:16.698241 1751 rpcz_store.cc:276] 0115 14:52:15.035001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.035114 (+ 113us) service_pool.cc:225] Handling call
0115 14:52:16.674118 (+1639004us) ts_tablet_manager.cc:1145] Deleting tablet 70425c25919b4d0b999450210e1cb33f
0115 14:52:16.674126 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.678483 (+ 4357us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.681285 (+ 2802us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.698125 (+ 16840us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1638843}
I20250115 14:52:16.699633 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 70425c25919b4d0b999450210e1cb33f (table default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]) successfully deleted
I20250115 14:52:16.699746 1168 tablet_bootstrap.cc:654] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.700039 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1265) took 1665 ms. Trace:
I20250115 14:52:16.700148 1751 rpcz_store.cc:276] 0115 14:52:15.034875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.035484 (+ 609us) service_pool.cc:225] Handling call
0115 14:52:16.699924 (+1664440us) ts_tablet_manager.cc:1145] Deleting tablet 5cb47732109e46d68e5af02ed52cb6d1
0115 14:52:16.699934 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.700022 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1663726}
W20250115 14:52:16.700706 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 5cb47732109e46d68e5af02ed52cb6d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cb47732109e46d68e5af02ed52cb6d1 already in progress: creating tablet
I20250115 14:52:16.701042 1751 tablet_replica.cc:331] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.701237 1751 raft_consensus.cc:2238] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.701403 1751 raft_consensus.cc:2267] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.701908 1751 ts_tablet_manager.cc:1905] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.703544 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6564555965984a1099d5d0525cb9883d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.704233 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6564555965984a1099d5d0525cb9883d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.707837 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_97011305bbd548d9bc1d1e2ea2730781"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.708384 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97011305bbd548d9bc1d1e2ea2730781 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.709415 1167 ts_tablet_manager.cc:1918] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.709517 1167 log.cc:1198] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/9f4475441e4a467a8d593b742d2a5b80
I20250115 14:52:16.709527 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b02e318b61ed4745bf6bf686cd720090" } new_table_name: "default.Table_e4daab9d83b344c584a35fbe8f39256b" modify_external_catalogs: true
I20250115 14:52:16.711067 1167 ts_tablet_manager.cc:1939] T 9f4475441e4a467a8d593b742d2a5b80 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:16.712141 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_1365cebf9380460289484fcc7b729f58" } new_table_name: "default.Table_970796a9329c4a71b73df0f226ae224e" modify_external_catalogs: true
I20250115 14:52:16.713312 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1247) took 2046 ms. Trace:
W20250115 14:52:16.713698 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 47342.8us
I20250115 14:52:16.713387 1167 rpcz_store.cc:276] 0115 14:52:14.667309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:14.870037 (+202728us) service_pool.cc:225] Handling call
0115 14:52:16.580121 (+1710084us) ts_tablet_manager.cc:1145] Deleting tablet 9f4475441e4a467a8d593b742d2a5b80
0115 14:52:16.580130 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.598792 (+ 18662us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.709307 (+110515us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.713296 (+ 3989us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1687588}
I20250115 14:52:16.715137 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 9f4475441e4a467a8d593b742d2a5b80 (table default.Table_c763759bb01d449cbd6e2f851b303ec5 [id=4f4b4f8aa6364221b4a35c423a5631aa]) successfully deleted
I20250115 14:52:16.715478 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1257) took 1385 ms. Trace:
I20250115 14:52:16.715590 1167 rpcz_store.cc:276] 0115 14:52:15.330493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.330608 (+ 115us) service_pool.cc:225] Handling call
0115 14:52:16.715363 (+1384755us) ts_tablet_manager.cc:1145] Deleting tablet 00844e77811b4cbfb7240176055e2966
0115 14:52:16.715376 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.715464 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1384250}
W20250115 14:52:16.716017 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 00844e77811b4cbfb7240176055e2966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00844e77811b4cbfb7240176055e2966 already in progress: creating tablet
I20250115 14:52:16.716387 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1258) took 1385 ms. Trace:
I20250115 14:52:16.716500 1167 rpcz_store.cc:276] 0115 14:52:15.330980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.331080 (+ 100us) service_pool.cc:225] Handling call
0115 14:52:16.716273 (+1385193us) ts_tablet_manager.cc:1145] Deleting tablet 2b26b9b1caf748aba7d36a04fbba2a80
0115 14:52:16.716282 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.716372 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1385029}
W20250115 14:52:16.716953 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2b26b9b1caf748aba7d36a04fbba2a80 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b26b9b1caf748aba7d36a04fbba2a80 already in progress: creating tablet
I20250115 14:52:16.717285 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1259) took 1386 ms. Trace:
I20250115 14:52:16.717391 1167 rpcz_store.cc:276] 0115 14:52:15.331364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.331466 (+ 102us) service_pool.cc:225] Handling call
0115 14:52:16.717196 (+1385730us) ts_tablet_manager.cc:1145] Deleting tablet 1020437ffddf49cc846fcd3311892b46
0115 14:52:16.717205 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.717272 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1385612}
W20250115 14:52:16.717761 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 1020437ffddf49cc846fcd3311892b46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1020437ffddf49cc846fcd3311892b46 already in progress: creating tablet
I20250115 14:52:16.718114 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1260) took 1386 ms. Trace:
I20250115 14:52:16.718218 1167 rpcz_store.cc:276] 0115 14:52:15.331801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.331838 (+ 37us) service_pool.cc:225] Handling call
0115 14:52:16.718028 (+1386190us) ts_tablet_manager.cc:1145] Deleting tablet 5cb47732109e46d68e5af02ed52cb6d1
0115 14:52:16.718036 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.718100 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1386049}
W20250115 14:52:16.718573 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 5cb47732109e46d68e5af02ed52cb6d1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cb47732109e46d68e5af02ed52cb6d1 already in progress: creating tablet
I20250115 14:52:16.718859 1167 tablet_replica.cc:331] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:16.720023 1751 ts_tablet_manager.cc:1918] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.720309 1167 raft_consensus.cc:2238] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:16.719798 1168 tablet_bootstrap.cc:492] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:16.720790 1168 ts_tablet_manager.cc:1397] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.004s
I20250115 14:52:16.720791 1751 log.cc:1198] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/86c04fa23e7c40a5b5e07852e5d74a27
I20250115 14:52:16.721146 1751 ts_tablet_manager.cc:1939] T 86c04fa23e7c40a5b5e07852e5d74a27 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.720655 1167 raft_consensus.cc:2267] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.721526 1168 raft_consensus.cc:357] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.721738 1168 raft_consensus.cc:383] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.721814 1168 raft_consensus.cc:738] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.722007 1168 consensus_queue.cc:260] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.722504 1168 ts_tablet_manager.cc:1428] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:16.720626 1162 tablet_bootstrap.cc:492] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:16.723337 1162 ts_tablet_manager.cc:1397] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.005s
I20250115 14:52:16.724946 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1270) took 1637 ms. Trace:
I20250115 14:52:16.725415 1167 ts_tablet_manager.cc:1905] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.726154 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2cb376913356450792ee862605884f0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:16.727226 2549 raft_consensus.cc:491] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.727337 2549 raft_consensus.cc:513] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.727900 2549 leader_election.cc:290] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:16.728551 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00e6cb19641f49e599275a5a5fc272fb consensus state has no leader
W20250115 14:52:16.728727 847 leader_election.cc:343] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6e2d55a4e32448fbb98f704dbf1657b9
W20250115 14:52:16.728929 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cb376913356450792ee862605884f0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.729301 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2d55a4e32448fbb98f704dbf1657b9" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.730587 1751 rpcz_store.cc:276] 0115 14:52:15.087795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.089603 (+ 1808us) service_pool.cc:225] Handling call
0115 14:52:16.700999 (+1611396us) ts_tablet_manager.cc:1145] Deleting tablet 86c04fa23e7c40a5b5e07852e5d74a27
0115 14:52:16.701008 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.704988 (+ 3980us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.719915 (+ 14927us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.724927 (+ 5012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1611208}
W20250115 14:52:16.730860 848 leader_election.cc:343] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.724049 1162 raft_consensus.cc:357] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.730957 848 leader_election.cc:304] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.731024 1162 raft_consensus.cc:383] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.728924 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b91939a04ebe4b688718c8f720121009"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:16.731175 2549 raft_consensus.cc:2743] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.731081 1162 raft_consensus.cc:738] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
W20250115 14:52:16.731488 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b91939a04ebe4b688718c8f720121009 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.731468 1162 consensus_queue.cc:260] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.731894 1751 tablet_replica.cc:331] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.732061 1751 raft_consensus.cc:2238] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.732215 1751 raft_consensus.cc:2267] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.732769 1751 ts_tablet_manager.cc:1905] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.730981 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 86c04fa23e7c40a5b5e07852e5d74a27 (table default.Table_702932472ce94b91b4535e0015a38be1 [id=04c521c944614e03be02fcb2ee8499c2]) successfully deleted
I20250115 14:52:16.735795 1162 ts_tablet_manager.cc:1428] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.012s user 0.000s sys 0.001s
I20250115 14:52:16.737130 1162 tablet_bootstrap.cc:492] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:16.737639 2549 raft_consensus.cc:491] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.737756 2549 raft_consensus.cc:513] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.738312 2549 leader_election.cc:290] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.739097 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_69731e3d99894dfa97e738208c012651"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.739554 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69731e3d99894dfa97e738208c012651 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.740554 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 09adb34bf9a3464798b9f28c50d6fc11 (table default.Table_f9271e15235f4b92b6f54d3decfb8a4e [id=37297c9e757d411daac43cbbea813e0c]). Expected version 1 got 0
I20250115 14:52:16.741367 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 2 got 0
I20250115 14:52:16.741459 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9915ea9ed9b749c8957dd5fb5152b5e5" } modify_external_catalogs: true
I20250115 14:52:16.741549 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bf648bb656634def8189fb514a07e144 (table default.Table_fd84fd3d47974e05a580fae352838c20 [id=a00115ecf6954f83b3118654f98e629d]). Expected version 1 got 0
I20250115 14:52:16.741582 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9915ea9ed9b749c8957dd5fb5152b5e5" } modify_external_catalogs: true
I20250115 14:52:16.741874 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 04daad461f0b433ba635df87439b5bc1 (table default.Table_c7251196621d49aa966653f5a456c7eb [id=289c1cdb743b49b5893f551afab62d43]). Expected version 1 got 0
I20250115 14:52:16.742000 447 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a5e97279185541b992340c24ede01656 (table default.Table_a8391687956e4259a71808f267cb4ff0 [id=361ed20953ab4e79b1112ae76b95a423]). Expected version 3 got 0
I20250115 14:52:16.747009 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_50430a4659d544eba42d4e0e1cef37da" } modify_external_catalogs: true
I20250115 14:52:16.747169 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_50430a4659d544eba42d4e0e1cef37da" } modify_external_catalogs: true
I20250115 14:52:16.747910 1168 tablet_bootstrap.cc:492] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:16.748838 2839 raft_consensus.cc:491] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.748970 2839 raft_consensus.cc:513] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.749598 2839 leader_election.cc:290] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.750579 1164 ts_tablet_manager.cc:1428] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.064s user 0.002s sys 0.000s
I20250115 14:52:16.750905 1164 tablet_bootstrap.cc:492] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:16.752691 1162 tablet_bootstrap.cc:654] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.753608 882 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9f3f6965e244febd08a89555f41024 (DEFAULT_TABLE table=default.Table_e5b03e341bbd4d11af2f2a999e4ffbcc [id=818b7ebbd73444d68df91a117bc5bf9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.753894 890 tablet_service.cc:1467] Processing CreateTablet for tablet 84941bbd2fa14404b9ca7b3a28c02e48 (DEFAULT_TABLE table=default.Table_6bb430490bfb46a18988ae5c2f2707a2 [id=8eaa8ac3cf554c8fbdf1753f36c5b585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.753947 884 tablet_service.cc:1467] Processing CreateTablet for tablet 52f279b07e364fb4a52171b9c13fac6a (DEFAULT_TABLE table=default.Table_230e0c86726c489cb19444c0ebea8a9f [id=63ef7e20900546ca9b2865b26d7f42d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.754202 1013 tablet_service.cc:1467] Processing CreateTablet for tablet d5ae350015c944f38479b5ddb1086b01 (DEFAULT_TABLE table=default.Table_186b04ca0bf6418895c667086df34188 [id=68cd9bf8efb2402fbdfd9b34d051393c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.754251 880 tablet_service.cc:1467] Processing CreateTablet for tablet d5ae350015c944f38479b5ddb1086b01 (DEFAULT_TABLE table=default.Table_186b04ca0bf6418895c667086df34188 [id=68cd9bf8efb2402fbdfd9b34d051393c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.754550 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ae350015c944f38479b5ddb1086b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.754508 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 84941bbd2fa14404b9ca7b3a28c02e48 (DEFAULT_TABLE table=default.Table_6bb430490bfb46a18988ae5c2f2707a2 [id=8eaa8ac3cf554c8fbdf1753f36c5b585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.754570 883 tablet_service.cc:1467] Processing CreateTablet for tablet d5d7c143a3e74192bf419135cb03bd99 (DEFAULT_TABLE table=default.Table_714851f5d34f456093fb3c60acb57e94 [id=89a00d8bb05847f4b870dbc66121d675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.754787 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84941bbd2fa14404b9ca7b3a28c02e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.754904 893 tablet_service.cc:1467] Processing CreateTablet for tablet 47a9b88d84274d829f3987858184f062 (DEFAULT_TABLE table=default.Table_d328c98aff614867b35565e6d7aecad8 [id=d4fcaf17c0a44f7e84118d67e8e08776]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.755240 877 tablet_service.cc:1467] Processing CreateTablet for tablet 353aed1b5d0c4acca0fe88b2e64a4170 (DEFAULT_TABLE table=default.Table_c3229d9207b641aaacbf3a4544824974 [id=4785247757d84f2e9fd9da24d671c9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.755606 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9f3f6965e244febd08a89555f41024 (DEFAULT_TABLE table=default.Table_e5b03e341bbd4d11af2f2a999e4ffbcc [id=818b7ebbd73444d68df91a117bc5bf9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.755918 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f3f6965e244febd08a89555f41024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.760090 1014 tablet_service.cc:1467] Processing CreateTablet for tablet d5d7c143a3e74192bf419135cb03bd99 (DEFAULT_TABLE table=default.Table_714851f5d34f456093fb3c60acb57e94 [id=89a00d8bb05847f4b870dbc66121d675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.760481 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 52f279b07e364fb4a52171b9c13fac6a (DEFAULT_TABLE table=default.Table_230e0c86726c489cb19444c0ebea8a9f [id=63ef7e20900546ca9b2865b26d7f42d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.760751 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52f279b07e364fb4a52171b9c13fac6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.764533 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 353aed1b5d0c4acca0fe88b2e64a4170 (DEFAULT_TABLE table=default.Table_c3229d9207b641aaacbf3a4544824974 [id=4785247757d84f2e9fd9da24d671c9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.764853 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 353aed1b5d0c4acca0fe88b2e64a4170. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.767390 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f3f6965e244febd08a89555f41024. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.767581 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 353aed1b5d0c4acca0fe88b2e64a4170. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.769699 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 47a9b88d84274d829f3987858184f062 (DEFAULT_TABLE table=default.Table_d328c98aff614867b35565e6d7aecad8 [id=d4fcaf17c0a44f7e84118d67e8e08776]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.769819 1168 tablet_bootstrap.cc:654] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.770016 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a9b88d84274d829f3987858184f062. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.770476 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a9b88d84274d829f3987858184f062. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.772401 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5d7c143a3e74192bf419135cb03bd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.773029 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84941bbd2fa14404b9ca7b3a28c02e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.776149 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:16.776402 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ae350015c944f38479b5ddb1086b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.776754 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52f279b07e364fb4a52171b9c13fac6a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.777223 983 leader_election.cc:343] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.777472 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5d7c143a3e74192bf419135cb03bd99. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.780515 847 leader_election.cc:343] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: a300a76b8f7942dea648a20cbb3d5368
I20250115 14:52:16.780747 2549 raft_consensus.cc:491] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.780869 2549 raft_consensus.cc:513] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.781438 2549 leader_election.cc:290] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.782640 750 tablet_service.cc:1467] Processing CreateTablet for tablet d5ae350015c944f38479b5ddb1086b01 (DEFAULT_TABLE table=default.Table_186b04ca0bf6418895c667086df34188 [id=68cd9bf8efb2402fbdfd9b34d051393c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.783035 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ae350015c944f38479b5ddb1086b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.783205 2403 raft_consensus.cc:491] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.783308 2403 raft_consensus.cc:513] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.783871 2403 leader_election.cc:290] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.784373 742 tablet_service.cc:1467] Processing CreateTablet for tablet 84941bbd2fa14404b9ca7b3a28c02e48 (DEFAULT_TABLE table=default.Table_6bb430490bfb46a18988ae5c2f2707a2 [id=8eaa8ac3cf554c8fbdf1753f36c5b585]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.784670 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84941bbd2fa14404b9ca7b3a28c02e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.786398 747 tablet_service.cc:1467] Processing CreateTablet for tablet 47a9b88d84274d829f3987858184f062 (DEFAULT_TABLE table=default.Table_d328c98aff614867b35565e6d7aecad8 [id=d4fcaf17c0a44f7e84118d67e8e08776]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.786679 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47a9b88d84274d829f3987858184f062. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.788482 743 tablet_service.cc:1467] Processing CreateTablet for tablet d5d7c143a3e74192bf419135cb03bd99 (DEFAULT_TABLE table=default.Table_714851f5d34f456093fb3c60acb57e94 [id=89a00d8bb05847f4b870dbc66121d675]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.788815 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5d7c143a3e74192bf419135cb03bd99. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.792449 737 tablet_service.cc:1467] Processing CreateTablet for tablet 52f279b07e364fb4a52171b9c13fac6a (DEFAULT_TABLE table=default.Table_230e0c86726c489cb19444c0ebea8a9f [id=63ef7e20900546ca9b2865b26d7f42d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.792614 741 tablet_service.cc:1467] Processing CreateTablet for tablet 353aed1b5d0c4acca0fe88b2e64a4170 (DEFAULT_TABLE table=default.Table_c3229d9207b641aaacbf3a4544824974 [id=4785247757d84f2e9fd9da24d671c9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.792758 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52f279b07e364fb4a52171b9c13fac6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.793000 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398ac8af619c427287b3964443173864" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.793670 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 353aed1b5d0c4acca0fe88b2e64a4170. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.796380 754 tablet_service.cc:1514] Processing DeleteTablet for tablet 325cd3fd8b4d447998822163e575e758 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.796401 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a300a76b8f7942dea648a20cbb3d5368" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.796723 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd30604ca05461da4e3a5e3eb10d982 (DEFAULT_TABLE table=default.Table_69be76f118b247d7b98ab4ec45332da0 [id=8f72ed3e7d684f2886ce0068f3bce379]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.736205 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ee552d095ba643fd8ffde9fc50ca565c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:16.796999 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dd30604ca05461da4e3a5e3eb10d982. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.798336 2831 raft_consensus.cc:491] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.798488 2831 raft_consensus.cc:513] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.799105 2831 leader_election.cc:290] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.800462 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d67ddf01e34541bb795755e03c8249" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:16.792449 748 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9f3f6965e244febd08a89555f41024 (DEFAULT_TABLE table=default.Table_e5b03e341bbd4d11af2f2a999e4ffbcc [id=818b7ebbd73444d68df91a117bc5bf9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.801028 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9f3f6965e244febd08a89555f41024. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.801111 983 leader_election.cc:343] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:16.801726 847 leader_election.cc:343] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 398ac8af619c427287b3964443173864
I20250115 14:52:16.802419 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd30604ca05461da4e3a5e3eb10d982 (DEFAULT_TABLE table=default.Table_69be76f118b247d7b98ab4ec45332da0 [id=8f72ed3e7d684f2886ce0068f3bce379]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.802805 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dd30604ca05461da4e3a5e3eb10d982. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.804472 2865 raft_consensus.cc:491] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.804585 2865 raft_consensus.cc:513] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:16.804713 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee552d095ba643fd8ffde9fc50ca565c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.805130 2865 leader_election.cc:290] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:16.805271 848 leader_election.cc:343] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.805383 848 leader_election.cc:304] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:16.805478 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 325cd3fd8b4d447998822163e575e758 because the tablet was not found. No further retry: Not found: Tablet not found: 325cd3fd8b4d447998822163e575e758
I20250115 14:52:16.805823 2748 raft_consensus.cc:2743] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:16.805860 848 leader_election.cc:343] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.805931 848 leader_election.cc:304] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.806094 2748 raft_consensus.cc:2743] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.806291 1042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:16.806646 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf648bb656634def8189fb514a07e144" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:16.810246 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 86c04fa23e7c40a5b5e07852e5d74a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.810480 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4c2707915e4e97ad6c66cb249c6d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6afbdd0b585e4e7e82067dce061924d0) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.810724 1023 tablet_service.cc:1467] Processing CreateTablet for tablet 3dd30604ca05461da4e3a5e3eb10d982 (DEFAULT_TABLE table=default.Table_69be76f118b247d7b98ab4ec45332da0 [id=8f72ed3e7d684f2886ce0068f3bce379]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.811120 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3dd30604ca05461da4e3a5e3eb10d982. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.812719 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 363433d4b13d4bee9d5bf1f97db0e404 (DEFAULT_TABLE table=default.Table_0e1d19fe704640f9a3cf90b94324f31b [id=58a168d8fc6c4904acb2469b2517339c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.813059 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363433d4b13d4bee9d5bf1f97db0e404. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.813169 710 leader_election.cc:343] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Not found: Tablet not found: 9f4475441e4a467a8d593b742d2a5b80
W20250115 14:52:16.813325 710 leader_election.cc:343] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 9f4475441e4a467a8d593b742d2a5b80
I20250115 14:52:16.813387 710 leader_election.cc:304] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:16.813702 710 leader_election.cc:304] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:16.814906 893 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5331d0aa3e4f278912ef9a840f6710 (DEFAULT_TABLE table=default.Table_c0788d0ac02f4683aa546d7c0682b59d [id=e3eb1fbbd94e4fae9fa8c42330f3b20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.814970 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet e6d67ddf01e34541bb795755e03c8249 (table default.Table_d495ddf9033f421e9d6cc7c06d08c8f2 [id=cf83ec5e8f3c4a0d92bdd962fbba9a67]). Expected version 1 got 0
I20250115 14:52:16.815136 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet d0f3f6de654d4765b1d7a381f93b0c43 (table default.Table_c118c528732d4868a96f871f5ab7e2bb [id=e7e8720c2f454d0e842cae5c2d32e5a0]). Expected version 1 got 0
I20250115 14:52:16.815219 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5331d0aa3e4f278912ef9a840f6710. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.815337 453 catalog_manager.cc:5526] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 reported cstate change: term changed from 0 to 1, leader changed from <none> to c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "c238c037a1c74725a001e2b4a1ff4304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: HEALTHY } } }
I20250115 14:52:16.815672 746 tablet_service.cc:1467] Processing CreateTablet for tablet 363433d4b13d4bee9d5bf1f97db0e404 (DEFAULT_TABLE table=default.Table_0e1d19fe704640f9a3cf90b94324f31b [id=58a168d8fc6c4904acb2469b2517339c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.815711 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 (table default.Table_a3ebaf4214574dd68c619382cb749409 [id=c66ea162404c4eb0a012128385984399]). Expected version 1 got 0
I20250115 14:52:16.815989 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363433d4b13d4bee9d5bf1f97db0e404. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.819213 890 tablet_service.cc:1467] Processing CreateTablet for tablet 363433d4b13d4bee9d5bf1f97db0e404 (DEFAULT_TABLE table=default.Table_0e1d19fe704640f9a3cf90b94324f31b [id=58a168d8fc6c4904acb2469b2517339c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.819581 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 363433d4b13d4bee9d5bf1f97db0e404. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.821827 748 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5331d0aa3e4f278912ef9a840f6710 (DEFAULT_TABLE table=default.Table_c0788d0ac02f4683aa546d7c0682b59d [id=e3eb1fbbd94e4fae9fa8c42330f3b20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.822216 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5331d0aa3e4f278912ef9a840f6710. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.824558 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5331d0aa3e4f278912ef9a840f6710 (DEFAULT_TABLE table=default.Table_c0788d0ac02f4683aa546d7c0682b59d [id=e3eb1fbbd94e4fae9fa8c42330f3b20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.824970 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a5331d0aa3e4f278912ef9a840f6710. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.826038 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.828339 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d67ddf01e34541bb795755e03c8249" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.828536 1751 ts_tablet_manager.cc:1918] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:16.828658 1751 log.cc:1198] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/4ee8b63cc21145d8993b058d8c6d23c7
I20250115 14:52:16.829015 1751 ts_tablet_manager.cc:1939] T 4ee8b63cc21145d8993b058d8c6d23c7 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:16.829700 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1300) took 1579 ms. Trace:
I20250115 14:52:16.829802 1751 rpcz_store.cc:276] 0115 14:52:15.253167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.275272 (+ 22105us) service_pool.cc:225] Handling call
0115 14:52:16.731848 (+1456576us) ts_tablet_manager.cc:1145] Deleting tablet 4ee8b63cc21145d8993b058d8c6d23c7
0115 14:52:16.731858 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.762236 (+ 30378us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.828462 (+ 66226us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.829657 (+ 1195us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1456417}
I20250115 14:52:16.830215 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 4ee8b63cc21145d8993b058d8c6d23c7 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]) successfully deleted
I20250115 14:52:16.830305 750 tablet_service.cc:1467] Processing CreateTablet for tablet 022844ce88b44b5f9639690c3976f6e6 (DEFAULT_TABLE table=default.Table_1dbc2a3b624144dcaf2ea044013159bf [id=2e7bbe54cc1c4eccb8de1ff81f9a4bb4]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:16.830574 847 consensus_peers.cc:487] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d -> Peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Couldn't send request to peer c238c037a1c74725a001e2b4a1ff4304. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36de948083a946eba1a96f276e4f4f91. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:16.830629 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 022844ce88b44b5f9639690c3976f6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.831758 890 tablet_service.cc:1467] Processing CreateTablet for tablet 022844ce88b44b5f9639690c3976f6e6 (DEFAULT_TABLE table=default.Table_1dbc2a3b624144dcaf2ea044013159bf [id=2e7bbe54cc1c4eccb8de1ff81f9a4bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.832084 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 022844ce88b44b5f9639690c3976f6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.836787 1167 ts_tablet_manager.cc:1918] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:16.836911 1167 log.cc:1198] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/70425c25919b4d0b999450210e1cb33f
I20250115 14:52:16.837235 1167 ts_tablet_manager.cc:1939] T 70425c25919b4d0b999450210e1cb33f P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:16.837302 2839 raft_consensus.cc:491] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.837432 2839 raft_consensus.cc:513] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.837986 2839 leader_election.cc:290] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.838596 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e910c387e6a24caf95e14b57c8b83e6f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:16.839107 983 leader_election.cc:343] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:16.840557 984 leader_election.cc:343] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.840677 984 leader_election.cc:304] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:16.841048 984 leader_election.cc:343] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.841154 984 leader_election.cc:304] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.841396 2831 raft_consensus.cc:2743] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.841559 2831 raft_consensus.cc:2743] T e6d67ddf01e34541bb795755e03c8249 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.844475 2403 raft_consensus.cc:2743] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.844619 2403 raft_consensus.cc:2743] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.848047 775 raft_consensus.cc:3054] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.851231 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e910c387e6a24caf95e14b57c8b83e6f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.854223 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1261) took 1522 ms. Trace:
I20250115 14:52:16.854312 1167 rpcz_store.cc:276] 0115 14:52:15.332201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.332311 (+ 110us) service_pool.cc:225] Handling call
0115 14:52:16.718810 (+1386499us) ts_tablet_manager.cc:1145] Deleting tablet 70425c25919b4d0b999450210e1cb33f
0115 14:52:16.718828 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.748779 (+ 29951us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.826379 (+ 77600us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.854204 (+ 27825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1386372}
I20250115 14:52:16.855773 1167 tablet_replica.cc:331] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:16.857384 1167 raft_consensus.cc:2238] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.857524 1167 raft_consensus.cc:2267] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.857992 1167 ts_tablet_manager.cc:1905] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.858393 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 2955ecdd45364dc2934ba96a9a6c8e8b
I20250115 14:52:16.862038 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e619d4aabff245f4b46f5d03cb9a856d
I20250115 14:52:16.862572 451 master_service.cc:946] ReplaceTablet: received request to replace tablet e663111811b34a578bc344c59be8649e from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:16.862669 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 022844ce88b44b5f9639690c3976f6e6 (DEFAULT_TABLE table=default.Table_1dbc2a3b624144dcaf2ea044013159bf [id=2e7bbe54cc1c4eccb8de1ff81f9a4bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.862843 2849 tablet.cc:1721] T 1b52fb251b0a48c2a8a4e07d55d22d3f P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:16.863107 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 022844ce88b44b5f9639690c3976f6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.864566 2774 tablet.cc:1721] T 1b52fb251b0a48c2a8a4e07d55d22d3f P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:16.865737 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1308) took 1249 ms. Trace:
I20250115 14:52:16.865846 1751 rpcz_store.cc:276] 0115 14:52:15.616693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.617236 (+ 543us) service_pool.cc:225] Handling call
0115 14:52:16.865587 (+1248351us) ts_tablet_manager.cc:1145] Deleting tablet 266521a877ba4fd2bc4914e500f65eae
0115 14:52:16.865598 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.865696 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1248180}
I20250115 14:52:16.866087 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet b6fb388c17044f5eabce9996ebbf4ff6
I20250115 14:52:16.866674 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_913e9383036c49f985fe9b1167c65efc" } modify_external_catalogs: true
I20250115 14:52:16.866838 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_913e9383036c49f985fe9b1167c65efc" } modify_external_catalogs: true
I20250115 14:52:16.868891 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1309) took 1252 ms. Trace:
I20250115 14:52:16.868988 1751 rpcz_store.cc:276] 0115 14:52:15.616835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.617440 (+ 605us) service_pool.cc:225] Handling call
0115 14:52:16.868721 (+1251281us) ts_tablet_manager.cc:1145] Deleting tablet a3b0bb374dc54533b7e8a5e5369c2f39
0115 14:52:16.868731 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.868855 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1251148}
I20250115 14:52:16.869622 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet de27f6c600b840c2b00a881c5ef038ae
I20250115 14:52:16.870308 1164 tablet_bootstrap.cc:654] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.870251 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7bc6f9ea70464e1cb86a77b5fb2f48dc" } new_table_name: "default.Table_9f71b01040004e2fa17677f7bf61cdf6" modify_external_catalogs: true
I20250115 14:52:16.871729 1168 tablet_bootstrap.cc:492] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:16.871866 1168 ts_tablet_manager.cc:1397] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.000s
I20250115 14:52:16.871969 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_d56c0d67f1514a0d8167c4a83162cbec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:16.872074 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1310) took 1253 ms. Trace:
I20250115 14:52:16.872161 1751 rpcz_store.cc:276] 0115 14:52:15.618299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.618467 (+ 168us) service_pool.cc:225] Handling call
0115 14:52:16.871941 (+1253474us) ts_tablet_manager.cc:1145] Deleting tablet 2923dd60927d49f5b0ca09438cfd4ce3
0115 14:52:16.871951 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.872030 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1253327}
W20250115 14:52:16.872437 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d56c0d67f1514a0d8167c4a83162cbec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.876822 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 2955ecdd45364dc2934ba96a9a6c8e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.877213 883 tablet_service.cc:1467] Processing CreateTablet for tablet 1b94a8c2462f49fe8902ab14ab3758c7 (DEFAULT_TABLE table=default.Table_793b55bcf9ea422588897663c768d387 [id=2db107850442422381b0900e5f9f820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.877342 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 70425c25919b4d0b999450210e1cb33f (table default.Table_b0a8c96252014e9298515a1f5ff14410 [id=1da36b8fd2074945badcd7d2f8eb0ecd]) successfully deleted
I20250115 14:52:16.877559 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b94a8c2462f49fe8902ab14ab3758c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.877664 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 266521a877ba4fd2bc4914e500f65eae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 266521a877ba4fd2bc4914e500f65eae already in progress: creating tablet
I20250115 14:52:16.877662 881 tablet_service.cc:1467] Processing CreateTablet for tablet 157bcd73adef45f3a628352599ac0da7 (DEFAULT_TABLE table=default.Table_892d26a61a5242e293e5fb8cff0d798c [id=d4a1b86ce12342e08f9514950f605c5b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:16.877851 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a3b0bb374dc54533b7e8a5e5369c2f39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3b0bb374dc54533b7e8a5e5369c2f39 already in progress: creating tablet
I20250115 14:52:16.877952 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157bcd73adef45f3a628352599ac0da7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.877990 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2923dd60927d49f5b0ca09438cfd4ce3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2923dd60927d49f5b0ca09438cfd4ce3 already in progress: creating tablet
I20250115 14:52:16.878952 880 tablet_service.cc:1514] Processing DeleteTablet for tablet b6fb388c17044f5eabce9996ebbf4ff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.878984 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 8c33907919374a6fbf9e7175c7c9a485 (DEFAULT_TABLE table=default.Table_10d8c601efc24a47b14b76858704b724 [id=d12776269f4045f78a988b1d8a56e691]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.879141 889 tablet_service.cc:1467] Processing CreateTablet for tablet d6965f175f7f4236920e06d71e986fb2 (DEFAULT_TABLE table=default.Table_8e77605904c9425cae44eed3f031d5f6 [id=c9fb22bcba6b4a7c8e9e8a36bb49021f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.879357 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c33907919374a6fbf9e7175c7c9a485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.879323 1023 tablet_service.cc:1467] Processing CreateTablet for tablet eedf1594322642c2a176d14442b0a99e (DEFAULT_TABLE table=default.Table_4bdcbfbff073498cb6257975ee190669 [id=ef4d858da4ad4b37a20df13fdd6ae92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.879426 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6965f175f7f4236920e06d71e986fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.879642 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eedf1594322642c2a176d14442b0a99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.879860 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet b6fb388c17044f5eabce9996ebbf4ff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.880343 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 157bcd73adef45f3a628352599ac0da7 (DEFAULT_TABLE table=default.Table_892d26a61a5242e293e5fb8cff0d798c [id=d4a1b86ce12342e08f9514950f605c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.880662 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157bcd73adef45f3a628352599ac0da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.882515 891 tablet_service.cc:1467] Processing CreateTablet for tablet bdea7d18b7984339a1d0bb1d59e6dce2 (DEFAULT_TABLE table=default.Table_c3dbb94e26f1439ba0bb576d883ec848 [id=0ba6de6dc23d4a078b54026c3f36bc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.882807 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdea7d18b7984339a1d0bb1d59e6dce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.884423 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet e619d4aabff245f4b46f5d03cb9a856d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.884755 877 tablet_service.cc:1467] Processing CreateTablet for tablet 8c33907919374a6fbf9e7175c7c9a485 (DEFAULT_TABLE table=default.Table_10d8c601efc24a47b14b76858704b724 [id=d12776269f4045f78a988b1d8a56e691]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.885018 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c33907919374a6fbf9e7175c7c9a485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.878952 892 tablet_service.cc:1514] Processing DeleteTablet for tablet de27f6c600b840c2b00a881c5ef038ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.885963 884 tablet_service.cc:1467] Processing CreateTablet for tablet eedf1594322642c2a176d14442b0a99e (DEFAULT_TABLE table=default.Table_4bdcbfbff073498cb6257975ee190669 [id=ef4d858da4ad4b37a20df13fdd6ae92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.886219 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eedf1594322642c2a176d14442b0a99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.887727 893 tablet_service.cc:1514] Processing DeleteTablet for tablet e619d4aabff245f4b46f5d03cb9a856d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:16.888833 1168 raft_consensus.cc:357] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.889010 1168 raft_consensus.cc:383] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:16.889062 1168 raft_consensus.cc:738] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:16.889248 1168 consensus_queue.cc:260] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.889441 2549 raft_consensus.cc:491] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.889542 2549 raft_consensus.cc:513] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.889724 1168 ts_tablet_manager.cc:1428] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20250115 14:52:16.890134 2549 leader_election.cc:290] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.890955 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:16.891153 1046 raft_consensus.cc:2463] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 0.
I20250115 14:52:16.891628 847 leader_election.cc:304] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.891871 2748 raft_consensus.cc:2798] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:16.891841 1011 tablet_service.cc:1467] Processing CreateTablet for tablet bdea7d18b7984339a1d0bb1d59e6dce2 (DEFAULT_TABLE table=default.Table_c3dbb94e26f1439ba0bb576d883ec848 [id=0ba6de6dc23d4a078b54026c3f36bc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.891929 2748 raft_consensus.cc:491] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:16.891979 2748 raft_consensus.cc:3054] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.892176 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdea7d18b7984339a1d0bb1d59e6dce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.894400 1016 tablet_service.cc:1467] Processing CreateTablet for tablet 1b94a8c2462f49fe8902ab14ab3758c7 (DEFAULT_TABLE table=default.Table_793b55bcf9ea422588897663c768d387 [id=2db107850442422381b0900e5f9f820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.894498 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet de27f6c600b840c2b00a881c5ef038ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.894770 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b94a8c2462f49fe8902ab14ab3758c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.896067 1168 tablet_bootstrap.cc:492] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:16.897678 1168 tablet_bootstrap.cc:654] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:16.898558 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2955ecdd45364dc2934ba96a9a6c8e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.899140 755 tablet_service.cc:1514] Processing DeleteTablet for tablet de27f6c600b840c2b00a881c5ef038ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.899165 737 tablet_service.cc:1467] Processing CreateTablet for tablet 157bcd73adef45f3a628352599ac0da7 (DEFAULT_TABLE table=default.Table_892d26a61a5242e293e5fb8cff0d798c [id=d4a1b86ce12342e08f9514950f605c5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.899236 747 tablet_service.cc:1467] Processing CreateTablet for tablet d6965f175f7f4236920e06d71e986fb2 (DEFAULT_TABLE table=default.Table_8e77605904c9425cae44eed3f031d5f6 [id=c9fb22bcba6b4a7c8e9e8a36bb49021f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.899492 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 157bcd73adef45f3a628352599ac0da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.899669 755 tablet_service.cc:1467] Processing CreateTablet for tablet bdea7d18b7984339a1d0bb1d59e6dce2 (DEFAULT_TABLE table=default.Table_c3dbb94e26f1439ba0bb576d883ec848 [id=0ba6de6dc23d4a078b54026c3f36bc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.899848 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6965f175f7f4236920e06d71e986fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.900017 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdea7d18b7984339a1d0bb1d59e6dce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.904392 754 tablet_service.cc:1514] Processing DeleteTablet for tablet b6fb388c17044f5eabce9996ebbf4ff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.904899 746 tablet_service.cc:1467] Processing CreateTablet for tablet 1b94a8c2462f49fe8902ab14ab3758c7 (DEFAULT_TABLE table=default.Table_793b55bcf9ea422588897663c768d387 [id=2db107850442422381b0900e5f9f820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.905045 768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.905216 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b94a8c2462f49fe8902ab14ab3758c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.906324 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet de27f6c600b840c2b00a881c5ef038ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de27f6c600b840c2b00a881c5ef038ae already in progress: creating tablet
W20250115 14:52:16.906483 848 leader_election.cc:343] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:16.906550 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet b6fb388c17044f5eabce9996ebbf4ff6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6fb388c17044f5eabce9996ebbf4ff6 already in progress: creating tablet
W20250115 14:52:16.906719 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 2955ecdd45364dc2934ba96a9a6c8e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2955ecdd45364dc2934ba96a9a6c8e8b already in progress: creating tablet
I20250115 14:52:16.908934 447 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 246720dbb4ed4216ac87fb4a3df79e2c (table default.Table_daaa2755fe9d4f54ae6ff33ae452ec05 [id=8c0229e6f6884ea7be3417678bb27377]). Expected version 1 got 0
I20250115 14:52:16.909111 447 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet ac4d850b85e948939008e5275d42e3cc (table default.Table_172dcf8ef25b424ca247c507a87c8d0f [id=427b1c6d3389420288bb30b1a4cc8b4f]). Expected version 1 got 0
I20250115 14:52:16.909235 748 tablet_service.cc:1467] Processing CreateTablet for tablet 96dc9f234dce48478a41e7f86d86c73a (DEFAULT_TABLE table=default.Table_14bb1fc9753d4ab987aab9744d650965 [id=f618ef27e1d94ada90c4d9e0e6147f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.909570 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96dc9f234dce48478a41e7f86d86c73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.909847 882 tablet_service.cc:1467] Processing CreateTablet for tablet 71aab76b5dbe44c8ad722a8b4d99d0a2 (DEFAULT_TABLE table=default.Table_ae26bf8b34aa4bd2a498fb5959363aa5 [id=a455094c186b463489b7dd85bdb973ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.910751 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71aab76b5dbe44c8ad722a8b4d99d0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.910677 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9c1c6dca23d34c0090f29d79b5d879fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.911182 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c1c6dca23d34c0090f29d79b5d879fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.912413 750 tablet_service.cc:1514] Processing DeleteTablet for tablet e619d4aabff245f4b46f5d03cb9a856d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.914979 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1311) took 1296 ms. Trace:
I20250115 14:52:16.915130 1751 rpcz_store.cc:276] 0115 14:52:15.618425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.618640 (+ 215us) service_pool.cc:225] Handling call
0115 14:52:16.914838 (+1296198us) ts_tablet_manager.cc:1145] Deleting tablet a5018e358d234ea28023da76a8c61a86
0115 14:52:16.914850 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.914943 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1257620}
I20250115 14:52:16.915665 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1316) took 1158 ms. Trace:
I20250115 14:52:16.915807 1751 rpcz_store.cc:276] 0115 14:52:15.757692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.757794 (+ 102us) service_pool.cc:225] Handling call
0115 14:52:16.915549 (+1157755us) ts_tablet_manager.cc:1145] Deleting tablet b7c6c8d5c58142c0af3e726ea57207b9
0115 14:52:16.915558 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.915633 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1157531}
I20250115 14:52:16.916252 1751 tablet_replica.cc:331] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:16.916887 1751 raft_consensus.cc:2238] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:16.917045 441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:40227: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250115 14:52:16.917974 1751 raft_consensus.cc:2267] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.918977 1751 ts_tablet_manager.cc:1905] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:16.920758 984 leader_election.cc:343] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.920902 984 leader_election.cc:304] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.921130 2839 raft_consensus.cc:2743] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:16.922778 2748 raft_consensus.cc:513] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.922924 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 2955ecdd45364dc2934ba96a9a6c8e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.923418 2748 leader_election.cc:290] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:16.924115 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304"
I20250115 14:52:16.924338 1046 raft_consensus.cc:3054] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:16.926749 1046 raft_consensus.cc:2463] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c86158cc2641f182ea50a06634592d in term 1.
I20250115 14:52:16.926978 743 tablet_service.cc:1467] Processing CreateTablet for tablet 8c33907919374a6fbf9e7175c7c9a485 (DEFAULT_TABLE table=default.Table_10d8c601efc24a47b14b76858704b724 [id=d12776269f4045f78a988b1d8a56e691]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.927351 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c33907919374a6fbf9e7175c7c9a485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.930186 741 tablet_service.cc:1467] Processing CreateTablet for tablet eedf1594322642c2a176d14442b0a99e (DEFAULT_TABLE table=default.Table_4bdcbfbff073498cb6257975ee190669 [id=ef4d858da4ad4b37a20df13fdd6ae92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.930518 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eedf1594322642c2a176d14442b0a99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.935061 847 leader_election.cc:304] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
I20250115 14:52:16.937043 2748 raft_consensus.cc:2798] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:16.937232 2748 raft_consensus.cc:695] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Becoming Leader. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Running, Role: LEADER
I20250115 14:52:16.937492 2748 consensus_queue.cc:237] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.942175 742 tablet_service.cc:1467] Processing CreateTablet for tablet 71aab76b5dbe44c8ad722a8b4d99d0a2 (DEFAULT_TABLE table=default.Table_ae26bf8b34aa4bd2a498fb5959363aa5 [id=a455094c186b463489b7dd85bdb973ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.943426 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71aab76b5dbe44c8ad722a8b4d99d0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.949221 1012 tablet_service.cc:1467] Processing CreateTablet for tablet d6965f175f7f4236920e06d71e986fb2 (DEFAULT_TABLE table=default.Table_8e77605904c9425cae44eed3f031d5f6 [id=c9fb22bcba6b4a7c8e9e8a36bb49021f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.950209 1167 ts_tablet_manager.cc:1918] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.950522 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 71aab76b5dbe44c8ad722a8b4d99d0a2 (DEFAULT_TABLE table=default.Table_ae26bf8b34aa4bd2a498fb5959363aa5 [id=a455094c186b463489b7dd85bdb973ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.951164 1167 log.cc:1198] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/e244689cef6844e2935becd265eecad0
I20250115 14:52:16.951608 1167 ts_tablet_manager.cc:1939] T e244689cef6844e2935becd265eecad0 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:16.952332 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca883d907214c27bfc119ef9dd687d5 (DEFAULT_TABLE table=default.Table_61919c493ca64b90818d51b3ee940d8b [id=4247e872dedb43608d11770aa5e6d3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.952740 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 96dc9f234dce48478a41e7f86d86c73a (DEFAULT_TABLE table=default.Table_14bb1fc9753d4ab987aab9744d650965 [id=f618ef27e1d94ada90c4d9e0e6147f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.952934 449 catalog_manager.cc:5526] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:16.953382 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 2 got 0
I20250115 14:52:16.953737 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 246720dbb4ed4216ac87fb4a3df79e2c (table default.Table_daaa2755fe9d4f54ae6ff33ae452ec05 [id=8c0229e6f6884ea7be3417678bb27377]). Expected version 1 got 0
I20250115 14:52:16.954028 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 5df0250d08214a9783c392071126ed4a (table default.Table_90c834d5ef634e6aa797dd220e3d5faa [id=8ffbe34c2a11417f8ca50dbdd627bf22]). Expected version 2 got 0
I20250115 14:52:16.954317 449 catalog_manager.cc:5526] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:16.954674 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 04daad461f0b433ba635df87439b5bc1 (table default.Table_c7251196621d49aa966653f5a456c7eb [id=289c1cdb743b49b5893f551afab62d43]). Expected version 1 got 0
I20250115 14:52:16.954842 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 (table default.Table_a3ebaf4214574dd68c619382cb749409 [id=c66ea162404c4eb0a012128385984399]). Expected version 1 got 0
I20250115 14:52:16.954952 449 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet bf648bb656634def8189fb514a07e144 (table default.Table_fd84fd3d47974e05a580fae352838c20 [id=a00115ecf6954f83b3118654f98e629d]). Expected version 1 got 0
I20250115 14:52:16.954236 891 tablet_service.cc:1467] Processing CreateTablet for tablet 96dc9f234dce48478a41e7f86d86c73a (DEFAULT_TABLE table=default.Table_14bb1fc9753d4ab987aab9744d650965 [id=f618ef27e1d94ada90c4d9e0e6147f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.955650 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96dc9f234dce48478a41e7f86d86c73a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:16.953727 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet a5018e358d234ea28023da76a8c61a86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5018e358d234ea28023da76a8c61a86 already in progress: creating tablet
I20250115 14:52:16.954604 881 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca883d907214c27bfc119ef9dd687d5 (DEFAULT_TABLE table=default.Table_61919c493ca64b90818d51b3ee940d8b [id=4247e872dedb43608d11770aa5e6d3a0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:16.956233 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b7c6c8d5c58142c0af3e726ea57207b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7c6c8d5c58142c0af3e726ea57207b9 already in progress: creating tablet
I20250115 14:52:16.956609 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca883d907214c27bfc119ef9dd687d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.959591 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 248cadcdef8b4e99923cdbca39690685 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:51:59 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.959930 750 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca883d907214c27bfc119ef9dd687d5 (DEFAULT_TABLE table=default.Table_61919c493ca64b90818d51b3ee940d8b [id=4247e872dedb43608d11770aa5e6d3a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:16.960480 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca883d907214c27bfc119ef9dd687d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:16.960844 755 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f4475441e4a467a8d593b742d2a5b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:16.961072 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
I20250115 14:52:16.960408 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_89ac4cd90d424ff9b3d332264f554d0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:16.961982 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89ac4cd90d424ff9b3d332264f554d0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:16.960403 2872 tablet.cc:1721] T 1b52fb251b0a48c2a8a4e07d55d22d3f P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:16.966212 2839 raft_consensus.cc:491] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:16.966363 2839 raft_consensus.cc:513] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:16.967022 2839 leader_election.cc:290] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:16.967396 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60e68b411364ada9b89879012d862ea" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:16.967521 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f60e68b411364ada9b89879012d862ea" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:16.967831 984 leader_election.cc:343] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:16.968026 983 leader_election.cc:343] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.968142 983 leader_election.cc:304] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:16.968551 1166 tablet_replica.cc:331] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.968772 1166 raft_consensus.cc:2238] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.968926 1166 raft_consensus.cc:2267] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.969591 1166 ts_tablet_manager.cc:1905] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.968544 2839 raft_consensus.cc:2743] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:16.974571 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e619d4aabff245f4b46f5d03cb9a856d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e619d4aabff245f4b46f5d03cb9a856d already in progress: creating tablet
W20250115 14:52:16.986032 848 leader_election.cc:343] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:16.986680 1166 ts_tablet_manager.cc:1918] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:16.986800 1166 log.cc:1198] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/248cadcdef8b4e99923cdbca39690685
I20250115 14:52:16.987556 1166 ts_tablet_manager.cc:1939] T 248cadcdef8b4e99923cdbca39690685 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:16.988660 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 248cadcdef8b4e99923cdbca39690685 (table default.Table_b64c63a30a1c4f328986efc44022083b [id=5d393887a5e54536a4139752211d21a1]) successfully deleted
I20250115 14:52:16.988529 1166 tablet_replica.cc:331] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:16.989084 1166 raft_consensus.cc:2238] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:16.989452 1166 raft_consensus.cc:2267] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:16.993832 1166 ts_tablet_manager.cc:1905] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:16.997165 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet f60e68b411364ada9b89879012d862ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.998482 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1265) took 1664 ms. Trace:
I20250115 14:52:17.000131 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet e180e37a1d5d46d78380323662b80f85 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a676807ef38485ea49b0e92f5dc9862) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:16.999707 1167 rpcz_store.cc:276] 0115 14:52:15.334213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.334272 (+ 59us) service_pool.cc:225] Handling call
0115 14:52:16.855724 (+1521452us) ts_tablet_manager.cc:1145] Deleting tablet e244689cef6844e2935becd265eecad0
0115 14:52:16.855734 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.874783 (+ 19049us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.950119 (+ 75336us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:16.998216 (+ 48097us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1521284}
I20250115 14:52:17.001318 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet c6cb8dfa7c42480bbd139d5acf059a57 with delete_type TABLET_DATA_DELETED (replaced by tablet 70167f772ef246f2b1b5fdb380d6f4da) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.001696 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6965f175f7f4236920e06d71e986fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.002100 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96dc9f234dce48478a41e7f86d86c73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.002507 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71aab76b5dbe44c8ad722a8b4d99d0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.002755 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ca883d907214c27bfc119ef9dd687d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:17.003060 848 consensus_peers.cc:487] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:17.013217 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 57aecd22f97c41618f90d14c2f53a7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.014838 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet e244689cef6844e2935becd265eecad0 (table default.Table_54e0718fb03d45629ac03649aacf44ba [id=1cdf416758254c679c3c785b101c65e2]) successfully deleted
I20250115 14:52:17.014889 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2d55a4e32448fbb98f704dbf1657b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:07 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.015004 883 tablet_service.cc:1514] Processing DeleteTablet for tablet a300a76b8f7942dea648a20cbb3d5368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.015151 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd94e4a7fef44f5be462e7a77e8a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.015713 884 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f89ec0a5d68408c897f43ab39df62b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.015342 881 tablet_service.cc:1514] Processing DeleteTablet for tablet ded663ac1197454bbde3c4c2d526002d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.016527 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6308f68e41554470b7d1321c60d4eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.016754 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 398ac8af619c427287b3964443173864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.017035 1167 tablet_replica.cc:331] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:17.018990 1167 raft_consensus.cc:2238] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.021301 1167 raft_consensus.cc:2267] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.022413 1167 ts_tablet_manager.cc:1905] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.023661 2774 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250115 14:52:17.033581 1162 tablet_bootstrap.cc:492] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:17.034121 1162 ts_tablet_manager.cc:1397] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.297s user 0.009s sys 0.007s
I20250115 14:52:17.034896 1166 ts_tablet_manager.cc:1918] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.035005 1166 log.cc:1198] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/9f4475441e4a467a8d593b742d2a5b80
I20250115 14:52:17.035292 1166 ts_tablet_manager.cc:1939] T 9f4475441e4a467a8d593b742d2a5b80 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:17.036188 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 9f4475441e4a467a8d593b742d2a5b80 (table default.Table_c763759bb01d449cbd6e2f851b303ec5 [id=4f4b4f8aa6364221b4a35c423a5631aa]) successfully deleted
I20250115 14:52:17.036931 2872 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250115 14:52:17.038360 1751 ts_tablet_manager.cc:1918] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250115 14:52:17.043412 1167 ts_tablet_manager.cc:1918] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:17.043605 1751 log.cc:1198] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d13b4c0edc2c4fc9a1c527e1ff6a3d00
I20250115 14:52:17.044209 1751 ts_tablet_manager.cc:1939] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:17.044639 1167 log.cc:1198] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/4ee8b63cc21145d8993b058d8c6d23c7
I20250115 14:52:17.045109 1167 ts_tablet_manager.cc:1939] T 4ee8b63cc21145d8993b058d8c6d23c7 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:17.045079 2849 tablet.cc:1721] T bf648bb656634def8189fb514a07e144 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.045423 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1295) took 1522 ms. Trace:
I20250115 14:52:17.045491 1167 rpcz_store.cc:276] 0115 14:52:15.523791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.523982 (+ 191us) service_pool.cc:225] Handling call
0115 14:52:17.016983 (+1493001us) ts_tablet_manager.cc:1145] Deleting tablet 4ee8b63cc21145d8993b058d8c6d23c7
0115 14:52:17.016995 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.040680 (+ 23685us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.043333 (+ 2653us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.045409 (+ 2076us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1444357}
I20250115 14:52:17.045836 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 4ee8b63cc21145d8993b058d8c6d23c7 (table default.Table_478cc36362494028a43f0e71465e1834 [id=e6177a285e0c4339a6cfc4a0ae779eae]) successfully deleted
I20250115 14:52:17.049288 1164 tablet_bootstrap.cc:492] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.049420 1164 ts_tablet_manager.cc:1397] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.299s user 0.008s sys 0.029s
I20250115 14:52:17.050204 2872 tablet.cc:1721] T bf648bb656634def8189fb514a07e144 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.054438 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1321) took 1183 ms. Trace:
I20250115 14:52:17.054672 1751 rpcz_store.cc:276] 0115 14:52:15.871108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.998379 (+127271us) service_pool.cc:225] Handling call
0115 14:52:16.916187 (+917808us) ts_tablet_manager.cc:1145] Deleting tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00
0115 14:52:16.916195 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:16.948116 (+ 31921us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.038249 (+ 90133us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.054391 (+ 16142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":917488}
I20250115 14:52:17.055677 1751 tablet_replica.cc:331] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:17.055939 1751 raft_consensus.cc:2238] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.056046 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_0d800cc4760a43fc82ef693a68ef4928 [id=be99ec07535243a68c1b3ee1be0c62e7]) successfully deleted
I20250115 14:52:17.056159 1751 raft_consensus.cc:2267] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.056694 1751 ts_tablet_manager.cc:1905] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.057943 1164 raft_consensus.cc:357] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.058665 1164 raft_consensus.cc:383] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.058847 1164 raft_consensus.cc:738] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.059175 1164 consensus_queue.cc:260] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.064446 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1301) took 1124 ms. Trace:
I20250115 14:52:17.064539 1167 rpcz_store.cc:276] 0115 14:52:15.940766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:15.941082 (+ 316us) service_pool.cc:225] Handling call
0115 14:52:17.064274 (+1123192us) ts_tablet_manager.cc:1145] Deleting tablet 266521a877ba4fd2bc4914e500f65eae
0115 14:52:17.064301 (+ 27us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.064424 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1121961}
W20250115 14:52:17.065169 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 266521a877ba4fd2bc4914e500f65eae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 266521a877ba4fd2bc4914e500f65eae already in progress: creating tablet
I20250115 14:52:17.065240 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1303) took 1034 ms. Trace:
I20250115 14:52:17.065311 1167 rpcz_store.cc:276] 0115 14:52:16.030349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.030431 (+ 82us) service_pool.cc:225] Handling call
0115 14:52:17.065127 (+1034696us) ts_tablet_manager.cc:1145] Deleting tablet 2923dd60927d49f5b0ca09438cfd4ce3
0115 14:52:17.065137 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.065222 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1034505}
W20250115 14:52:17.065840 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2923dd60927d49f5b0ca09438cfd4ce3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2923dd60927d49f5b0ca09438cfd4ce3 already in progress: creating tablet
I20250115 14:52:17.066731 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1304) took 1035 ms. Trace:
I20250115 14:52:17.066812 1167 rpcz_store.cc:276] 0115 14:52:16.030883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.030961 (+ 78us) service_pool.cc:225] Handling call
0115 14:52:17.066634 (+1035673us) ts_tablet_manager.cc:1145] Deleting tablet a5018e358d234ea28023da76a8c61a86
0115 14:52:17.066643 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.066716 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035494}
I20250115 14:52:17.067158 1164 ts_tablet_manager.cc:1428] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
W20250115 14:52:17.067376 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a5018e358d234ea28023da76a8c61a86 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5018e358d234ea28023da76a8c61a86 already in progress: creating tablet
I20250115 14:52:17.067566 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1302) took 1037 ms. Trace:
I20250115 14:52:17.067637 1167 rpcz_store.cc:276] 0115 14:52:16.030078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.051167 (+ 21089us) service_pool.cc:225] Handling call
0115 14:52:17.067475 (+1016308us) ts_tablet_manager.cc:1145] Deleting tablet a3b0bb374dc54533b7e8a5e5369c2f39
0115 14:52:17.067484 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.067552 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1016132}
W20250115 14:52:17.067896 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet a3b0bb374dc54533b7e8a5e5369c2f39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3b0bb374dc54533b7e8a5e5369c2f39 already in progress: creating tablet
I20250115 14:52:17.068894 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1309) took 1001 ms. Trace:
I20250115 14:52:17.068972 1167 rpcz_store.cc:276] 0115 14:52:16.067560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.072348 (+ 4788us) service_pool.cc:225] Handling call
0115 14:52:17.068796 (+996448us) ts_tablet_manager.cc:1145] Deleting tablet b7c6c8d5c58142c0af3e726ea57207b9
0115 14:52:17.068805 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.068878 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":996266}
I20250115 14:52:17.069548 1164 tablet_bootstrap.cc:492] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:17.070683 1167 tablet_replica.cc:331] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:17.070871 1167 raft_consensus.cc:2238] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
W20250115 14:52:17.071077 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b7c6c8d5c58142c0af3e726ea57207b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7c6c8d5c58142c0af3e726ea57207b9 already in progress: creating tablet
I20250115 14:52:17.071187 1167 raft_consensus.cc:2267] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.071712 1167 ts_tablet_manager.cc:1905] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.073448 1164 tablet_bootstrap.cc:654] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.037994 1162 raft_consensus.cc:357] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.040877 2774 tablet.cc:1721] T bf648bb656634def8189fb514a07e144 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.074884 1162 raft_consensus.cc:383] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.075350 1162 raft_consensus.cc:738] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.076766 1751 ts_tablet_manager.cc:1918] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.076875 1751 log.cc:1198] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/0f19f0a73ca24a37a5ef8e19b86ccc3c
I20250115 14:52:17.077164 1751 ts_tablet_manager.cc:1939] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:17.076593 1162 consensus_queue.cc:260] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.079313 2849 tablet.cc:1721] T d7309ecb11d844848638c042c9bbe5f1 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250115 14:52:17.079437 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1329) took 1032 ms. Trace:
I20250115 14:52:17.079515 1751 rpcz_store.cc:276] 0115 14:52:16.046931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.047146 (+ 215us) service_pool.cc:225] Handling call
0115 14:52:17.055610 (+1008464us) ts_tablet_manager.cc:1145] Deleting tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c
0115 14:52:17.055621 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.065987 (+ 10366us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.076654 (+ 10667us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.079400 (+ 2746us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1062,"tablet-delete.queue_time_us":1008286}
I20250115 14:52:17.079933 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 0f19f0a73ca24a37a5ef8e19b86ccc3c (table default.Table_924719cc6f6a45fc99e425268591354c [id=b085d42488df4fee9c6da3f98b0a163a]) successfully deleted
I20250115 14:52:17.080334 1751 tablet_replica.cc:331] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:17.080499 1751 raft_consensus.cc:2238] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.080659 1751 raft_consensus.cc:2267] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.081146 1751 ts_tablet_manager.cc:1905] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.086870 1167 ts_tablet_manager.cc:1918] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20250115 14:52:17.086971 1167 log.cc:1198] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/d13b4c0edc2c4fc9a1c527e1ff6a3d00
I20250115 14:52:17.087282 1167 ts_tablet_manager.cc:1939] T d13b4c0edc2c4fc9a1c527e1ff6a3d00 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:17.089139 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet d13b4c0edc2c4fc9a1c527e1ff6a3d00 (table default.Table_0d800cc4760a43fc82ef693a68ef4928 [id=be99ec07535243a68c1b3ee1be0c62e7]) successfully deleted
I20250115 14:52:17.089709 1167 tablet_replica.cc:331] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:17.089890 1167 raft_consensus.cc:2238] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:17.090170 1167 pending_rounds.cc:62] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:17.090212 1167 pending_rounds.cc:69] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558405757878272 op_type: NO_OP noop_request { }
I20250115 14:52:17.090312 1167 raft_consensus.cc:2883] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:17.090382 1167 raft_consensus.cc:2267] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.090884 1167 ts_tablet_manager.cc:1905] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.092459 1162 ts_tablet_manager.cc:1428] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.058s user 0.004s sys 0.001s
I20250115 14:52:17.094388 1751 ts_tablet_manager.cc:1918] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:17.094504 1751 log.cc:1198] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/3c2ee5dfdef14ec092a2f9d4bfd295a6
I20250115 14:52:17.094830 1751 ts_tablet_manager.cc:1939] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:17.095839 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_ebfbdc698c8c448ca2e60757b1fd574c [id=471b029f1dba4dd087deb339bba02013]) successfully deleted
I20250115 14:52:17.100584 1162 tablet_bootstrap.cc:492] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:17.101454 1167 ts_tablet_manager.cc:1918] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:17.101563 1167 log.cc:1198] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/36de948083a946eba1a96f276e4f4f91
I20250115 14:52:17.101933 1167 ts_tablet_manager.cc:1939] T 36de948083a946eba1a96f276e4f4f91 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:17.102404 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 36de948083a946eba1a96f276e4f4f91 (table default.Table_89373c553f9c407a9e205bfe0d1acf99 [id=26c53728ab22430b95ca322543118a43]) successfully deleted
I20250115 14:52:17.102509 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1331) took 1001 ms. Trace:
I20250115 14:52:17.102595 1751 rpcz_store.cc:276] 0115 14:52:16.101096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.103523 (+ 2427us) service_pool.cc:225] Handling call
0115 14:52:17.097868 (+994345us) ts_tablet_manager.cc:1145] Deleting tablet 87026b6faf75437eac21dedab1f41ed4
0115 14:52:17.102359 (+ 4491us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.102466 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":994118}
W20250115 14:52:17.103003 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 87026b6faf75437eac21dedab1f41ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87026b6faf75437eac21dedab1f41ed4 already in progress: creating tablet
I20250115 14:52:17.103333 1751 tablet_replica.cc:331] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:17.103544 1751 raft_consensus.cc:2238] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.103713 1751 raft_consensus.cc:2267] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.104243 1751 ts_tablet_manager.cc:1905] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.106215 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet bf648bb656634def8189fb514a07e144 (table default.Table_fd84fd3d47974e05a580fae352838c20 [id=a00115ecf6954f83b3118654f98e629d]). Expected version 1 got 0
I20250115 14:52:17.108124 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e663111811b34a578bc344c59be8649e
W20250115 14:52:17.108234 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 651ea2c7d1d7447a9ac69b1b120f19cd consensus state has no leader
I20250115 14:52:17.109280 2831 raft_consensus.cc:491] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.109416 2831 raft_consensus.cc:513] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.109426 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_ca1016e060984b6eb5d54a07aad3e076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.109941 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca1016e060984b6eb5d54a07aad3e076 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.110051 2831 leader_election.cc:290] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:17.110821 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ce16fea4f9469c9fc06fa24f6554be" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:17.111385 983 leader_election.cc:343] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.112658 882 tablet_service.cc:1514] Processing DeleteTablet for tablet e663111811b34a578bc344c59be8649e with delete_type TABLET_DATA_DELETED (replaced by tablet dd2e8aa0012444a489087213f9111f19) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.112396 1167 tablet_replica.cc:331] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:17.113003 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5c321ec5b2c84f01a09fa02678c31b4b" } new_table_name: "default.Table_5ecbe7f38d8947bf94c32dbd3e335c39" modify_external_catalogs: true
I20250115 14:52:17.113584 1167 raft_consensus.cc:2238] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:17.114102 1167 raft_consensus.cc:2267] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.114739 1167 ts_tablet_manager.cc:1905] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.114861 1162 tablet_bootstrap.cc:654] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.115630 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet e663111811b34a578bc344c59be8649e with delete_type TABLET_DATA_DELETED (replaced by tablet dd2e8aa0012444a489087213f9111f19) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.116024 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c49eefbfc9ce404eb604fb265c6a64d0
I20250115 14:52:17.116130 1751 ts_tablet_manager.cc:1918] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.116329 2748 consensus_queue.cc:1035] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:17.116745 1751 log.cc:1198] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f400880835ab4efdb2af0be5553d20dc
I20250115 14:52:17.116720 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9f03b68742a7468d8e2f0e5e4af9f094"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:17.117069 1751 ts_tablet_manager.cc:1939] T f400880835ab4efdb2af0be5553d20dc P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:17.117192 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f03b68742a7468d8e2f0e5e4af9f094 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.118530 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet c49eefbfc9ce404eb604fb265c6a64d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.119251 2839 consensus_queue.cc:1035] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:17.120357 882 tablet_service.cc:1514] Processing DeleteTablet for tablet c49eefbfc9ce404eb604fb265c6a64d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.092732 768 raft_consensus.cc:3054] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:17.108275 455 catalog_manager.cc:5526] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c86158cc2641f182ea50a06634592d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "73c86158cc2641f182ea50a06634592d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } health_report { overall_health: UNKNOWN } } }
I20250115 14:52:17.122985 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet e49dc97eea474252a5e0ef4af6e015a9 (table default.Table_4f00ef2ef63d4f77929a245aa9bff035 [id=8502b5fe264f4fb5a89bd4a91949c90b]). Expected version 1 got 0
I20250115 14:52:17.110257 746 tablet_service.cc:1514] Processing DeleteTablet for tablet e663111811b34a578bc344c59be8649e with delete_type TABLET_DATA_DELETED (replaced by tablet dd2e8aa0012444a489087213f9111f19) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.111941 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ce16fea4f9469c9fc06fa24f6554be" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.109059 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7f8c179068074379b0e5baec32f2699b" } new_table_name: "default.Table_b1d70c03aeaf40ef839cccf69c196fe4" modify_external_catalogs: true
I20250115 14:52:17.116806 743 tablet_service.cc:1514] Processing DeleteTablet for tablet c49eefbfc9ce404eb604fb265c6a64d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.109094 451 catalog_manager.cc:6596] ReplaceTablet: tablet e663111811b34a578bc344c59be8649e deleted and replaced by tablet dd2e8aa0012444a489087213f9111f19
W20250115 14:52:17.126005 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03892c1eaf5a474dbba9ad35cbf0ec4d consensus state has no leader
I20250115 14:52:17.126178 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_52bce80521874a99b9965322de01bc4b" } new_table_name: "default.Table_a8b1aa7c820340108d648c2628acb4cc" modify_external_catalogs: true
I20250115 14:52:17.126286 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7aef87d795794fd4ae0d9a8776e8cd79" } new_table_name: "default.Table_68c4f36399df440cb374d87ff5a44fb0" modify_external_catalogs: true
W20250115 14:52:17.128991 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet e663111811b34a578bc344c59be8649e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e663111811b34a578bc344c59be8649e already in progress: creating tablet
W20250115 14:52:17.129211 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c49eefbfc9ce404eb604fb265c6a64d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49eefbfc9ce404eb604fb265c6a64d0 already in progress: creating tablet
W20250115 14:52:17.129470 984 leader_election.cc:343] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.129429 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1a6fffc23de14381b074fafe7e270032"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:17.129591 984 leader_election.cc:304] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:17.129877 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a6fffc23de14381b074fafe7e270032 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.130287 2831 raft_consensus.cc:2743] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.130509 450 master_service.cc:946] ReplaceTablet: received request to replace tablet 5e2ffb75f16c4244b795c21f8b89d32a from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:17.132519 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3" } modify_external_catalogs: true
I20250115 14:52:17.132663 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_42eb3a3adc4b4f7bb5b44e97bf98f4f3" } modify_external_catalogs: true
W20250115 14:52:17.135298 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36749313f40648ac8cec4ac7f3d97cf8 consensus state has no leader
I20250115 14:52:17.135715 2839 raft_consensus.cc:491] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.135865 2839 raft_consensus.cc:513] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.136512 2839 leader_election.cc:290] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:17.136571 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c6e6ffce5fa42f5a82f9b94982127e4 consensus state has no leader
I20250115 14:52:17.137028 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e180e37a1d5d46d78380323662b80f85" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:17.138093 983 leader_election.cc:343] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.138381 2839 raft_consensus.cc:491] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.138511 2839 raft_consensus.cc:513] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.139081 2839 leader_election.cc:290] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:17.139600 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9494c0c57be8499a883888d067c7db77" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:17.140157 983 leader_election.cc:343] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:17.148679 848 consensus_peers.cc:487] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:17.149380 1042 raft_consensus.cc:1270] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:17.149943 2835 consensus_queue.cc:1035] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:17.152998 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4980e328dcce4600a482147b7a9c0b30" } modify_external_catalogs: true
I20250115 14:52:17.153184 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4980e328dcce4600a482147b7a9c0b30" } modify_external_catalogs: true
I20250115 14:52:17.154417 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_498abe3411a94a3d8bbbf3c96571396c" } modify_external_catalogs: true
I20250115 14:52:17.154556 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_498abe3411a94a3d8bbbf3c96571396c" } modify_external_catalogs: true
W20250115 14:52:17.159906 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 328fbbc9e15b42b1ac5398d4cfec1ede consensus state has no leader
I20250115 14:52:17.160992 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_27c8babf31ef4f10b816e123b4efd9d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.161453 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27c8babf31ef4f10b816e123b4efd9d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.163154 1164 tablet_bootstrap.cc:492] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.163303 1164 ts_tablet_manager.cc:1397] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.095s user 0.004s sys 0.000s
I20250115 14:52:17.163432 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f400880835ab4efdb2af0be5553d20dc (table default.Table_38b74aa452d14ca5b22b7259f85af561 [id=ce47660a249c41c68f6b6b2c2c9cd485]) successfully deleted
I20250115 14:52:17.164086 454 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8cb55ab9a193424ab91010e11b559f39" } modify_external_catalogs: true
I20250115 14:52:17.164053 1164 raft_consensus.cc:357] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.164237 1164 raft_consensus.cc:383] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.164227 454 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8cb55ab9a193424ab91010e11b559f39" } modify_external_catalogs: true
I20250115 14:52:17.164281 1164 raft_consensus.cc:738] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.164230 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5e2ffb75f16c4244b795c21f8b89d32a
I20250115 14:52:17.164489 1164 consensus_queue.cc:260] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.164690 450 catalog_manager.cc:6596] ReplaceTablet: tablet 5e2ffb75f16c4244b795c21f8b89d32a deleted and replaced by tablet fc465e2526444b378af8ee978fff4d54
I20250115 14:52:17.165190 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2ffb75f16c4244b795c21f8b89d32a with delete_type TABLET_DATA_DELETED (replaced by tablet fc465e2526444b378af8ee978fff4d54) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.165263 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2ffb75f16c4244b795c21f8b89d32a with delete_type TABLET_DATA_DELETED (replaced by tablet fc465e2526444b378af8ee978fff4d54) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:17.165814 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b3daf6416ad847aeaa83087cc81c9239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3daf6416ad847aeaa83087cc81c9239 already in progress: creating tablet
I20250115 14:52:17.166059 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2ffb75f16c4244b795c21f8b89d32a with delete_type TABLET_DATA_DELETED (replaced by tablet fc465e2526444b378af8ee978fff4d54) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:17.166781 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7c3393ed28d242838d6f2f170184f7e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c3393ed28d242838d6f2f170184f7e8 already in progress: creating tablet
W20250115 14:52:17.167148 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5e2ffb75f16c4244b795c21f8b89d32a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e2ffb75f16c4244b795c21f8b89d32a already in progress: creating tablet
W20250115 14:52:17.167624 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 4968ce56c6b4457097880a8ea7f924d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4968ce56c6b4457097880a8ea7f924d8 already in progress: creating tablet
W20250115 14:52:17.168578 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7db35ff0383549de86230bb1eb89389f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db35ff0383549de86230bb1eb89389f already in progress: creating tablet
I20250115 14:52:17.171193 1751 tablet_replica.cc:331] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:17.171828 1751 raft_consensus.cc:2238] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.172468 1751 raft_consensus.cc:2267] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.173476 1751 ts_tablet_manager.cc:1905] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.181650 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_d79f3054627d46fe87697c3985a729ed [id=904e587a8da8478a8b6c9134502377d8] alter complete (version 1)
I20250115 14:52:17.182297 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_fd84fd3d47974e05a580fae352838c20 [id=a00115ecf6954f83b3118654f98e629d] alter complete (version 1)
I20250115 14:52:17.183323 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet c300667f775e40b5a65dce61f4863cae
I20250115 14:52:17.184098 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet c300667f775e40b5a65dce61f4863cae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.184531 743 tablet_service.cc:1514] Processing DeleteTablet for tablet c300667f775e40b5a65dce61f4863cae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:17.186676 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet c300667f775e40b5a65dce61f4863cae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c300667f775e40b5a65dce61f4863cae already in progress: creating tablet
I20250115 14:52:17.187289 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet eeb5ad85b4524994bba041dfb3fc8e1f
I20250115 14:52:17.187832 743 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb5ad85b4524994bba041dfb3fc8e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.188020 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb5ad85b4524994bba041dfb3fc8e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:17.188670 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet eeb5ad85b4524994bba041dfb3fc8e1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeb5ad85b4524994bba041dfb3fc8e1f already in progress: creating tablet
I20250115 14:52:17.188691 882 tablet_service.cc:1514] Processing DeleteTablet for tablet c300667f775e40b5a65dce61f4863cae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.188889 891 tablet_service.cc:1514] Processing DeleteTablet for tablet eeb5ad85b4524994bba041dfb3fc8e1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:17.189210 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34127, user_credentials={real_user=slave}} blocked reactor thread for 59754.1us
I20250115 14:52:17.189955 2908 tablet.cc:1721] T ac4d850b85e948939008e5275d42e3cc P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.190486 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet a5e97279185541b992340c24ede01656 (table default.Table_a8391687956e4259a71808f267cb4ff0 [id=361ed20953ab4e79b1112ae76b95a423]). Expected version 3 got 0
I20250115 14:52:17.192032 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9494c0c57be8499a883888d067c7db77" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.192735 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 828be3fd7b634d23b88aebf4633c41fe
I20250115 14:52:17.193754 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 828be3fd7b634d23b88aebf4633c41fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.193969 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 828be3fd7b634d23b88aebf4633c41fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.195336 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 828be3fd7b634d23b88aebf4633c41fe with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:17.196161 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 828be3fd7b634d23b88aebf4633c41fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 828be3fd7b634d23b88aebf4633c41fe already in progress: creating tablet
I20250115 14:52:17.196707 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e180e37a1d5d46d78380323662b80f85" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.197588 1162 tablet_bootstrap.cc:492] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:17.197764 1162 ts_tablet_manager.cc:1397] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.098s user 0.001s sys 0.004s
I20250115 14:52:17.198480 1162 raft_consensus.cc:357] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.198729 1162 raft_consensus.cc:383] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.198833 1162 raft_consensus.cc:738] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.198863 2835 raft_consensus.cc:491] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.199005 2835 raft_consensus.cc:513] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.199211 1162 consensus_queue.cc:260] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.199576 2835 leader_election.cc:290] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:17.199939 1162 ts_tablet_manager.cc:1428] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:17.200471 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd94e4a7fef44f5be462e7a77e8a123" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:17.200546 847 leader_election.cc:343] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 0fd94e4a7fef44f5be462e7a77e8a123
W20250115 14:52:17.200958 848 leader_election.cc:343] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.201045 848 leader_election.cc:304] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.201488 2835 raft_consensus.cc:2743] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:17.202482 984 consensus_peers.cc:487] T a5e97279185541b992340c24ede01656 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250115 14:52:17.202692 984 leader_election.cc:343] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.202852 984 leader_election.cc:304] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.202921 1162 tablet_bootstrap.cc:492] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
W20250115 14:52:17.203299 984 leader_election.cc:343] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.203444 984 leader_election.cc:304] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.203507 2915 raft_consensus.cc:2743] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.203753 2839 raft_consensus.cc:2743] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.204411 1168 tablet_bootstrap.cc:492] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:17.204540 1168 ts_tablet_manager.cc:1397] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.309s user 0.003s sys 0.000s
I20250115 14:52:17.205461 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_970796a9329c4a71b73df0f226ae224e [id=855145809143431d81790dd36147a77b] alter complete (version 4)
I20250115 14:52:17.205475 1168 raft_consensus.cc:357] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.205649 1168 raft_consensus.cc:383] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.205693 1168 raft_consensus.cc:738] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.205917 1168 consensus_queue.cc:260] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.206101 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 86c04fa23e7c40a5b5e07852e5d74a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.206306 1166 tablet_replica.cc:331] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:17.206498 1166 raft_consensus.cc:2238] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.206581 1168 ts_tablet_manager.cc:1428] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:17.206677 1166 raft_consensus.cc:2267] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.206845 1168 tablet_bootstrap.cc:492] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:17.208158 1167 ts_tablet_manager.cc:1918] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:17.208266 1167 log.cc:1198] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/3c2ee5dfdef14ec092a2f9d4bfd295a6
I20250115 14:52:17.208671 1167 ts_tablet_manager.cc:1939] T 3c2ee5dfdef14ec092a2f9d4bfd295a6 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:17.210153 1166 ts_tablet_manager.cc:1905] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.217085 1164 ts_tablet_manager.cc:1428] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.054s user 0.000s sys 0.002s
I20250115 14:52:17.217410 1164 tablet_bootstrap.cc:492] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:17.217839 452 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 2 got 0
I20250115 14:52:17.218647 2849 tablet.cc:1721] T 04daad461f0b433ba635df87439b5bc1 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.218811 2774 tablet.cc:1721] T 04daad461f0b433ba635df87439b5bc1 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.222776 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 48b15de9bf8943cdb9e5abd943484993 with delete_type TABLET_DATA_DELETED (replaced by tablet 1de35551e26643b3b1e570ab8b2f6f56) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.226745 1166 ts_tablet_manager.cc:1918] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.226786 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 2 got 0
I20250115 14:52:17.226876 1166 log.cc:1198] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/86c04fa23e7c40a5b5e07852e5d74a27
W20250115 14:52:17.229311 441 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 9c6e6ffce5fa42f5a82f9b94982127e4 (table default.Table_a8b1aa7c820340108d648c2628acb4cc [id=bb66ca33fd3b4396a49eb4a6b6f0848f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9c6e6ffce5fa42f5a82f9b94982127e4 consensus state has no leader
I20250115 14:52:17.229957 1166 ts_tablet_manager.cc:1939] T 86c04fa23e7c40a5b5e07852e5d74a27 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:17.230917 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 86c04fa23e7c40a5b5e07852e5d74a27 (table default.Table_702932472ce94b91b4535e0015a38be1 [id=04c521c944614e03be02fcb2ee8499c2]) successfully deleted
I20250115 14:52:17.232443 1162 tablet_bootstrap.cc:654] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.247774 774 raft_consensus.cc:3054] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:17.248711 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a5e97279185541b992340c24ede01656 (table default.Table_a8391687956e4259a71808f267cb4ff0 [id=361ed20953ab4e79b1112ae76b95a423]). Expected version 3 got 0
I20250115 14:52:17.248932 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 2 got 0
I20250115 14:52:17.249068 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 04daad461f0b433ba635df87439b5bc1 (table default.Table_c7251196621d49aa966653f5a456c7eb [id=289c1cdb743b49b5893f551afab62d43]). Expected version 1 got 0
I20250115 14:52:17.250936 2835 consensus_queue.cc:1035] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:17.259313 446 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_c7251196621d49aa966653f5a456c7eb [id=289c1cdb743b49b5893f551afab62d43] alter complete (version 1)
I20250115 14:52:17.249948 1751 ts_tablet_manager.cc:1918] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.259622 1751 log.cc:1198] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e910c387e6a24caf95e14b57c8b83e6f
I20250115 14:52:17.260797 1013 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ce16fea4f9469c9fc06fa24f6554be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.263989 1751 ts_tablet_manager.cc:1939] T e910c387e6a24caf95e14b57c8b83e6f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:17.264453 1162 tablet_bootstrap.cc:492] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:17.264619 1162 ts_tablet_manager.cc:1397] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.005s
I20250115 14:52:17.266790 452 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 2 got 0
I20250115 14:52:17.268530 1164 tablet_bootstrap.cc:654] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.273250 891 tablet_service.cc:1467] Processing CreateTablet for tablet c6a5233af1d24c6a87c18f90369f5580 (DEFAULT_TABLE table=default.Table_312f79187f444b58988d8cda414ce44c [id=956374c66254411bb297ef07db1ad782]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.273347 1013 tablet_service.cc:1467] Processing CreateTablet for tablet c6a5233af1d24c6a87c18f90369f5580 (DEFAULT_TABLE table=default.Table_312f79187f444b58988d8cda414ce44c [id=956374c66254411bb297ef07db1ad782]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.273638 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a5233af1d24c6a87c18f90369f5580. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.273638 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a5233af1d24c6a87c18f90369f5580. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.273581 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 298e9089381643be9556f43fa5be1d5a (DEFAULT_TABLE table=default.Table_5f71454d37804130bd571f372234edc7 [id=b2e86207d14d468cb2f4cee8805071e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.273891 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 298e9089381643be9556f43fa5be1d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.274251 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e910c387e6a24caf95e14b57c8b83e6f (table default.Table_74ce013274db42559be764fbed84c959 [id=12d6246d23134956a43c01cc54d29fe1]) successfully deleted
I20250115 14:52:17.274535 1751 tablet_replica.cc:331] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:17.274701 1751 raft_consensus.cc:2238] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.274827 1751 raft_consensus.cc:2267] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.275274 1751 ts_tablet_manager.cc:1905] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.276531 882 tablet_service.cc:1467] Processing CreateTablet for tablet 298e9089381643be9556f43fa5be1d5a (DEFAULT_TABLE table=default.Table_5f71454d37804130bd571f372234edc7 [id=b2e86207d14d468cb2f4cee8805071e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.276832 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 298e9089381643be9556f43fa5be1d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.279826 881 tablet_service.cc:1467] Processing CreateTablet for tablet 3986c11941da4cf5a69ec98847393f01 (DEFAULT_TABLE table=default.Table_def3b3bdfb824258849c1d6a35f228a8 [id=8f0d1862e5f04c04be3681c442e59867]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.280136 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3986c11941da4cf5a69ec98847393f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.282382 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 2 got 0
I20250115 14:52:17.283305 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 7b6ea13ef1884d66bd1e7655df3e925d from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:17.283821 746 tablet_service.cc:1467] Processing CreateTablet for tablet 298e9089381643be9556f43fa5be1d5a (DEFAULT_TABLE table=default.Table_5f71454d37804130bd571f372234edc7 [id=b2e86207d14d468cb2f4cee8805071e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.283901 743 tablet_service.cc:1467] Processing CreateTablet for tablet c6a5233af1d24c6a87c18f90369f5580 (DEFAULT_TABLE table=default.Table_312f79187f444b58988d8cda414ce44c [id=956374c66254411bb297ef07db1ad782]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.283973 775 raft_consensus.cc:3054] T 246720dbb4ed4216ac87fb4a3df79e2c P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:17.284199 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 298e9089381643be9556f43fa5be1d5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.285413 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3986c11941da4cf5a69ec98847393f01 (DEFAULT_TABLE table=default.Table_def3b3bdfb824258849c1d6a35f228a8 [id=8f0d1862e5f04c04be3681c442e59867]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.285720 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6a5233af1d24c6a87c18f90369f5580. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.285737 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3986c11941da4cf5a69ec98847393f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.286898 742 tablet_service.cc:1467] Processing CreateTablet for tablet 3986c11941da4cf5a69ec98847393f01 (DEFAULT_TABLE table=default.Table_def3b3bdfb824258849c1d6a35f228a8 [id=8f0d1862e5f04c04be3681c442e59867]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.287068 2549 raft_consensus.cc:491] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.287271 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3986c11941da4cf5a69ec98847393f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.287186 2549 raft_consensus.cc:513] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.287819 2549 leader_election.cc:290] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:17.287930 2549 raft_consensus.cc:491] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.288007 2549 raft_consensus.cc:513] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.288712 2549 leader_election.cc:290] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:17.289551 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f89ec0a5d68408c897f43ab39df62b1" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.290403 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 342bf8d4dc4244f982b1b63aaae0fb45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.290585 884 tablet_service.cc:1467] Processing CreateTablet for tablet e7f22d90cf4d4ee4a907ce4eff439bea (DEFAULT_TABLE table=default.Table_93b0e0aeb9414c3daff5c72ffe77258e [id=abecdb8d19a54da98c733d5da641e28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.290905 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f22d90cf4d4ee4a907ce4eff439bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.292065 1012 tablet_service.cc:1467] Processing CreateTablet for tablet e7f22d90cf4d4ee4a907ce4eff439bea (DEFAULT_TABLE table=default.Table_93b0e0aeb9414c3daff5c72ffe77258e [id=abecdb8d19a54da98c733d5da641e28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.292426 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f22d90cf4d4ee4a907ce4eff439bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.293447 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbf52515f654f0db478536121f5753b (DEFAULT_TABLE table=default.Table_740d21d5d42e459b8325b4cfaa1bf85f [id=b4e982218cfb4e30b92fea2646fe2385]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.295428 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fbf52515f654f0db478536121f5753b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.295428 741 tablet_service.cc:1467] Processing CreateTablet for tablet e7f22d90cf4d4ee4a907ce4eff439bea (DEFAULT_TABLE table=default.Table_93b0e0aeb9414c3daff5c72ffe77258e [id=abecdb8d19a54da98c733d5da641e28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.295554 750 tablet_service.cc:1467] Processing CreateTablet for tablet 69553d636b524bc4b7ad79fe844d8d70 (DEFAULT_TABLE table=default.Table_40404fbf248d43aeab53e7db74beb0c3 [id=339a81e43d81466eb7ff00d748fb2d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.295770 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f22d90cf4d4ee4a907ce4eff439bea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.296025 755 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbf52515f654f0db478536121f5753b (DEFAULT_TABLE table=default.Table_740d21d5d42e459b8325b4cfaa1bf85f [id=b4e982218cfb4e30b92fea2646fe2385]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.296342 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fbf52515f654f0db478536121f5753b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.297914 1168 tablet_bootstrap.cc:654] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.300521 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 3601449ddebd4a57a460557971a87ff8 (DEFAULT_TABLE table=default.Table_1401e274205941c5bd2a45b1edf58461 [id=df981b491b3a472590883156d94c9a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.300561 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 6d465f7cdf9f480cad7d69c547da7169 (DEFAULT_TABLE table=default.Table_6564555965984a1099d5d0525cb9883d [id=6f863725aaac4b27b3a6a3504d06d6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.300825 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3601449ddebd4a57a460557971a87ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.301370 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 69553d636b524bc4b7ad79fe844d8d70 (DEFAULT_TABLE table=default.Table_40404fbf248d43aeab53e7db74beb0c3 [id=339a81e43d81466eb7ff00d748fb2d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.301676 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69553d636b524bc4b7ad79fe844d8d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.304344 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 14d04d1833e941b7b792842be74e6465 (DEFAULT_TABLE table=default.Table_36b868736e9e46528bddc87ba5998ff8 [id=54b53910e9da4ff5a92b8b219cd539a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.304632 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d04d1833e941b7b792842be74e6465. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.306274 1011 tablet_service.cc:1467] Processing CreateTablet for tablet abed3c5605604fd1ac63710b50a5b69a (DEFAULT_TABLE table=default.Table_73c030f6c4c44f9bb54b65a8356b9c87 [id=5a3792d40ac94963b37fed2628be0d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.306392 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69553d636b524bc4b7ad79fe844d8d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.306850 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6754ab1e6865448fafba9318cd2ea307"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.307416 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6754ab1e6865448fafba9318cd2ea307 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.307857 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_e9987ad74ce74ebbb8467da42a76c854"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.308318 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9987ad74ce74ebbb8467da42a76c854 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.309314 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d465f7cdf9f480cad7d69c547da7169. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.310173 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6308f68e41554470b7d1321c60d4eee9" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.310170 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_19e166ae7bf948e394394e722ff61d97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.310546 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19e166ae7bf948e394394e722ff61d97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.311895 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_baa1a5a968444c6ba2e8094bf8cf3163" } new_table_name: "default.Table_03ec69a9efed48609c3c4c97b522a4d3" modify_external_catalogs: true
W20250115 14:52:17.312515 848 leader_election.cc:343] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.313213 877 tablet_service.cc:1467] Processing CreateTablet for tablet 6fbf52515f654f0db478536121f5753b (DEFAULT_TABLE table=default.Table_740d21d5d42e459b8325b4cfaa1bf85f [id=b4e982218cfb4e30b92fea2646fe2385]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.313547 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fbf52515f654f0db478536121f5753b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.313599 883 tablet_service.cc:1467] Processing CreateTablet for tablet 69553d636b524bc4b7ad79fe844d8d70 (DEFAULT_TABLE table=default.Table_40404fbf248d43aeab53e7db74beb0c3 [id=339a81e43d81466eb7ff00d748fb2d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.313880 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69553d636b524bc4b7ad79fe844d8d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.316565 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3601449ddebd4a57a460557971a87ff8 (DEFAULT_TABLE table=default.Table_1401e274205941c5bd2a45b1edf58461 [id=df981b491b3a472590883156d94c9a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.316825 893 tablet_service.cc:1467] Processing CreateTablet for tablet 14d04d1833e941b7b792842be74e6465 (DEFAULT_TABLE table=default.Table_36b868736e9e46528bddc87ba5998ff8 [id=54b53910e9da4ff5a92b8b219cd539a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.317023 892 tablet_service.cc:1467] Processing CreateTablet for tablet abed3c5605604fd1ac63710b50a5b69a (DEFAULT_TABLE table=default.Table_73c030f6c4c44f9bb54b65a8356b9c87 [id=5a3792d40ac94963b37fed2628be0d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.316833 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3601449ddebd4a57a460557971a87ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.317581 1164 tablet_bootstrap.cc:492] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.317714 1164 ts_tablet_manager.cc:1397] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.000s
I20250115 14:52:17.316535 890 tablet_service.cc:1467] Processing CreateTablet for tablet 6d465f7cdf9f480cad7d69c547da7169 (DEFAULT_TABLE table=default.Table_6564555965984a1099d5d0525cb9883d [id=6f863725aaac4b27b3a6a3504d06d6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.318298 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d465f7cdf9f480cad7d69c547da7169. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.318401 1164 raft_consensus.cc:357] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.318609 1164 raft_consensus.cc:383] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.318675 1164 raft_consensus.cc:738] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.318883 1164 consensus_queue.cc:260] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.319378 1164 ts_tablet_manager.cc:1428] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:17.319645 1164 tablet_bootstrap.cc:492] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:17.320377 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abed3c5605604fd1ac63710b50a5b69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.321105 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d04d1833e941b7b792842be74e6465. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.323216 1751 ts_tablet_manager.cc:1918] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.323304 1751 log.cc:1198] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/9494c0c57be8499a883888d067c7db77
W20250115 14:52:17.323408 848 leader_election.cc:343] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.323559 1751 ts_tablet_manager.cc:1939] T 9494c0c57be8499a883888d067c7db77 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:17.325703 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 126e01e9520544619f3a124586937fb5
I20250115 14:52:17.326481 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fea785c7867c44bd8271e1705fe4d238"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.327006 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fea785c7867c44bd8271e1705fe4d238 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.332962 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7b6ea13ef1884d66bd1e7655df3e925d
I20250115 14:52:17.333348 446 catalog_manager.cc:6596] ReplaceTablet: tablet 7b6ea13ef1884d66bd1e7655df3e925d deleted and replaced by tablet 6846983c1b5c456f90292d5f5b600426
I20250115 14:52:17.334661 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 126e01e9520544619f3a124586937fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.334757 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6ea13ef1884d66bd1e7655df3e925d with delete_type TABLET_DATA_DELETED (replaced by tablet 6846983c1b5c456f90292d5f5b600426) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.340915 1168 tablet_bootstrap.cc:492] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:17.341082 1168 ts_tablet_manager.cc:1397] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.134s user 0.002s sys 0.002s
I20250115 14:52:17.344002 1168 raft_consensus.cc:357] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.344197 1168 raft_consensus.cc:383] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.344249 1168 raft_consensus.cc:738] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.344455 1168 consensus_queue.cc:260] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.345484 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_555a230c62e04d098ec432545d739dba" } new_table_name: "default.Table_d380516807ea439c9eb4d8250e673f91" modify_external_catalogs: true
I20250115 14:52:17.343596 748 tablet_service.cc:1467] Processing CreateTablet for tablet 6d465f7cdf9f480cad7d69c547da7169 (DEFAULT_TABLE table=default.Table_6564555965984a1099d5d0525cb9883d [id=6f863725aaac4b27b3a6a3504d06d6cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.346506 1168 ts_tablet_manager.cc:1428] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250115 14:52:17.346969 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_765ae278c41246aca70aa158f4ee2a08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.347421 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_765ae278c41246aca70aa158f4ee2a08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.349459 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_8aae5406185442c5aadb27f03012605f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:17.349815 1168 tablet_bootstrap.cc:492] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:17.349898 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8aae5406185442c5aadb27f03012605f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.351253 1168 tablet_bootstrap.cc:654] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.352138 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5c58ab16f3b6495681b201cf7a5652c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.352542 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c58ab16f3b6495681b201cf7a5652c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.353650 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_95f70e728c46403fa5002e0f2a01b5b8" } modify_external_catalogs: true
I20250115 14:52:17.353792 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_95f70e728c46403fa5002e0f2a01b5b8" } modify_external_catalogs: true
I20250115 14:52:17.357465 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6ea13ef1884d66bd1e7655df3e925d with delete_type TABLET_DATA_DELETED (replaced by tablet 6846983c1b5c456f90292d5f5b600426) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.359557 741 tablet_service.cc:1467] Processing CreateTablet for tablet 14d04d1833e941b7b792842be74e6465 (DEFAULT_TABLE table=default.Table_36b868736e9e46528bddc87ba5998ff8 [id=54b53910e9da4ff5a92b8b219cd539a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.360548 746 tablet_service.cc:1467] Processing CreateTablet for tablet abed3c5605604fd1ac63710b50a5b69a (DEFAULT_TABLE table=default.Table_73c030f6c4c44f9bb54b65a8356b9c87 [id=5a3792d40ac94963b37fed2628be0d5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.361912 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 126e01e9520544619f3a124586937fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:17.362581 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b940dce7de14c1b844f8835d436a3d8 consensus state has no leader
I20250115 14:52:17.365986 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f9271e15235f4b92b6f54d3decfb8a4e" } new_table_name: "default.Table_2e56658df72047609417d02e32f69b18" modify_external_catalogs: true
W20250115 14:52:17.366891 847 leader_election.cc:343] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 6308f68e41554470b7d1321c60d4eee9
I20250115 14:52:17.366997 847 leader_election.cc:304] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.367115 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abed3c5605604fd1ac63710b50a5b69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.367159 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d100b9e08d3045d083f5800a8d651450" } new_table_name: "default.Table_3f6e2223f6d44b11881ceae5b8cc4afd" modify_external_catalogs: true
I20250115 14:52:17.368372 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ac31746e6f2f4855b46e06e0f7d95a16" } new_table_name: "default.Table_5811c6224c6d4b89b33f313200011b90" modify_external_catalogs: true
I20250115 14:52:17.370479 2549 raft_consensus.cc:2743] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:17.372843 847 leader_election.cc:343] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 4f89ec0a5d68408c897f43ab39df62b1
I20250115 14:52:17.372953 847 leader_election.cc:304] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.373293 2835 raft_consensus.cc:2743] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.377245 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d465f7cdf9f480cad7d69c547da7169. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.377781 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d04d1833e941b7b792842be74e6465. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.378397 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abed3c5605604fd1ac63710b50a5b69a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:17.382750 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 99449d50c2dd46159a69dc0af1de7a87 consensus state has no leader
I20250115 14:52:17.385181 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cd7cf126bef040e7aac0c1b681b4370d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.385766 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd7cf126bef040e7aac0c1b681b4370d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.388370 1168 tablet_bootstrap.cc:492] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:17.388499 1168 ts_tablet_manager.cc:1397] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.039s user 0.008s sys 0.006s
I20250115 14:52:17.389637 1168 raft_consensus.cc:357] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.389833 1168 raft_consensus.cc:383] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.389882 1168 raft_consensus.cc:738] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.390280 1168 consensus_queue.cc:260] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.390960 1168 ts_tablet_manager.cc:1428] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:17.393200 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 2 got 0
I20250115 14:52:17.394008 2835 raft_consensus.cc:491] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.394131 2835 raft_consensus.cc:513] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.394670 2835 leader_election.cc:290] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:17.395440 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_162eb94aa1674d21ae644021401a54e8" } modify_external_catalogs: true
I20250115 14:52:17.395594 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_162eb94aa1674d21ae644021401a54e8" } modify_external_catalogs: true
W20250115 14:52:17.396391 847 leader_election.cc:343] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 0cae4c9228c640e495507e8f7ad9f544
I20250115 14:52:17.397080 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_12f038b27f044196a4c4f37f1d246025"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.397603 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12f038b27f044196a4c4f37f1d246025 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.399266 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cae4c9228c640e495507e8f7ad9f544 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.401609 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_74825cfc55d54eab828b4eef26d5148e" } modify_external_catalogs: true
I20250115 14:52:17.401741 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_74825cfc55d54eab828b4eef26d5148e" } modify_external_catalogs: true
I20250115 14:52:17.406883 893 tablet_service.cc:1514] Processing DeleteTablet for tablet 18f23815196047628d562da7312b010f with delete_type TABLET_DATA_DELETED (replaced by tablet 05bb99f36ba649d8b607e9a95ab58ed0) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.407742 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7c37ffc6335846248f463334008c0fe5" } modify_external_catalogs: true
I20250115 14:52:17.407889 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7c37ffc6335846248f463334008c0fe5" } modify_external_catalogs: true
I20250115 14:52:17.409209 893 tablet_service.cc:1467] Processing CreateTablet for tablet 69ef11c26b91409093b1322e5d426208 (DEFAULT_TABLE table=default.Table_0cec3b5c1b834606bcb399747273ec4c [id=27cc4441615e4a7d9da2f1f035a1be41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.409603 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69ef11c26b91409093b1322e5d426208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.418097 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 69ef11c26b91409093b1322e5d426208 (DEFAULT_TABLE table=default.Table_0cec3b5c1b834606bcb399747273ec4c [id=27cc4441615e4a7d9da2f1f035a1be41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.418442 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69ef11c26b91409093b1322e5d426208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.425123 881 tablet_service.cc:1467] Processing CreateTablet for tablet e7de7f2b01a3474cb5bed484d8d9b01e (DEFAULT_TABLE table=default.Table_b91939a04ebe4b688718c8f720121009 [id=0c0c9685483749ad8d358f8fab77241f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.416945 1162 raft_consensus.cc:357] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.425519 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7de7f2b01a3474cb5bed484d8d9b01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.425719 1162 raft_consensus.cc:383] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.426529 1012 tablet_service.cc:1467] Processing CreateTablet for tablet e7de7f2b01a3474cb5bed484d8d9b01e (DEFAULT_TABLE table=default.Table_b91939a04ebe4b688718c8f720121009 [id=0c0c9685483749ad8d358f8fab77241f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.426935 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7de7f2b01a3474cb5bed484d8d9b01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.429718 1751 tablet_replica.cc:331] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:17.429980 1751 raft_consensus.cc:2238] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.430187 1751 raft_consensus.cc:2267] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.430730 1751 ts_tablet_manager.cc:1905] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.432415 1162 raft_consensus.cc:738] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.432629 1162 consensus_queue.cc:260] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.433094 1162 ts_tablet_manager.cc:1428] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20250115 14:52:17.433318 1162 tablet_bootstrap.cc:492] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:17.433881 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 9494c0c57be8499a883888d067c7db77 (table default.Table_6b196b51d11442e5be9d5a3a2f6f3484 [id=d6e71eafa9aa42d2bf53bf5c4bb82fd2]) successfully deleted
I20250115 14:52:17.435324 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 66107249af824a0d87e0de1219f5b9fa (DEFAULT_TABLE table=default.Table_928cab0a21204c4d82a7bca5e4a0c193 [id=eb7665f922e74dea81f16012d8b984ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.435664 890 tablet_service.cc:1467] Processing CreateTablet for tablet a3f636c8d2174702ab628551268724f5 (DEFAULT_TABLE table=default.Table_e8d3a3b353b14d8bb1f1c0afb200b824 [id=dcc82589e228441e8aff9ebb6fe5fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.436211 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3f636c8d2174702ab628551268724f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.439314 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 93fff736d93b4d638e6492aa1294df0d (DEFAULT_TABLE table=default.Table_25bedca633db44cfbf7fcc28e160d831 [id=47484b11237c4f3d90aad8262ff4bd8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.440527 892 tablet_service.cc:1467] Processing CreateTablet for tablet 66107249af824a0d87e0de1219f5b9fa (DEFAULT_TABLE table=default.Table_928cab0a21204c4d82a7bca5e4a0c193 [id=eb7665f922e74dea81f16012d8b984ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.440841 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66107249af824a0d87e0de1219f5b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.441640 891 tablet_service.cc:1467] Processing CreateTablet for tablet 93fff736d93b4d638e6492aa1294df0d (DEFAULT_TABLE table=default.Table_25bedca633db44cfbf7fcc28e160d831 [id=47484b11237c4f3d90aad8262ff4bd8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.441922 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93fff736d93b4d638e6492aa1294df0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.446978 755 tablet_service.cc:1467] Processing CreateTablet for tablet 3601449ddebd4a57a460557971a87ff8 (DEFAULT_TABLE table=default.Table_1401e274205941c5bd2a45b1edf58461 [id=df981b491b3a472590883156d94c9a6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.450115 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66107249af824a0d87e0de1219f5b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.451645 1162 tablet_bootstrap.cc:654] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.452261 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 126e01e9520544619f3a124586937fb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.452440 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b6ea13ef1884d66bd1e7655df3e925d with delete_type TABLET_DATA_DELETED (replaced by tablet 6846983c1b5c456f90292d5f5b600426) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.453284 1164 tablet_bootstrap.cc:654] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.453774 1012 tablet_service.cc:1467] Processing CreateTablet for tablet a3f636c8d2174702ab628551268724f5 (DEFAULT_TABLE table=default.Table_e8d3a3b353b14d8bb1f1c0afb200b824 [id=dcc82589e228441e8aff9ebb6fe5fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.454082 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3f636c8d2174702ab628551268724f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.454663 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93fff736d93b4d638e6492aa1294df0d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:17.456810 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 126e01e9520544619f3a124586937fb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 126e01e9520544619f3a124586937fb5 already in progress: creating tablet
W20250115 14:52:17.457001 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7b6ea13ef1884d66bd1e7655df3e925d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b6ea13ef1884d66bd1e7655df3e925d already in progress: creating tablet
I20250115 14:52:17.457427 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cae4c9228c640e495507e8f7ad9f544" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.458238 737 tablet_service.cc:1467] Processing CreateTablet for tablet 92551c13d9dd4408a3609df997184b69 (DEFAULT_TABLE table=default.Table_35647648e7e64b1382583fb35ae12dc9 [id=9a3309da08bd4a309286d14ce1fb8ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.458760 754 tablet_service.cc:1467] Processing CreateTablet for tablet 3bccb7b153f04a6d87edec011d878c0a (DEFAULT_TABLE table=default.Table_a2dac92e95de48bfb2af2bdd00b50c33 [id=4e4c188c02d443af99e85a602baa316c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.459224 739 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd0c62e3692490ab5c24b0f6f430870 (DEFAULT_TABLE table=default.Table_97011305bbd548d9bc1d1e2ea2730781 [id=04a97532b94a4b41a77418ca1861ac5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.459347 743 tablet_service.cc:1467] Processing CreateTablet for tablet 69ef11c26b91409093b1322e5d426208 (DEFAULT_TABLE table=default.Table_0cec3b5c1b834606bcb399747273ec4c [id=27cc4441615e4a7d9da2f1f035a1be41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.459646 749 tablet_service.cc:1467] Processing CreateTablet for tablet 368de0e78e334d82a588a9ed89dc4b27 (DEFAULT_TABLE table=default.Table_5beff98980534f9694b89e4fa022ab17 [id=65192124c9a64955af16569fd4320422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.459780 748 tablet_service.cc:1467] Processing CreateTablet for tablet e7de7f2b01a3474cb5bed484d8d9b01e (DEFAULT_TABLE table=default.Table_b91939a04ebe4b688718c8f720121009 [id=0c0c9685483749ad8d358f8fab77241f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.458284 756 tablet_service.cc:1467] Processing CreateTablet for tablet 9c672b93188548778a827cc745fe5aec (DEFAULT_TABLE table=default.Table_aac511b008a343a4a9b676f06ef31a5b [id=642161d0d6a0434f8c78aacf985c1eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.461854 750 tablet_service.cc:1467] Processing CreateTablet for tablet 66107249af824a0d87e0de1219f5b9fa (DEFAULT_TABLE table=default.Table_928cab0a21204c4d82a7bca5e4a0c193 [id=eb7665f922e74dea81f16012d8b984ab]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:17.462240 984 consensus_peers.cc:487] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:17.462271 747 tablet_service.cc:1467] Processing CreateTablet for tablet 93fff736d93b4d638e6492aa1294df0d (DEFAULT_TABLE table=default.Table_25bedca633db44cfbf7fcc28e160d831 [id=47484b11237c4f3d90aad8262ff4bd8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.462726 751 tablet_service.cc:1467] Processing CreateTablet for tablet e3fd8b80954c40a0b7679e1dfa6b4671 (DEFAULT_TABLE table=default.Table_75c1d7f0fd574f53a5b1a2853de03c30 [id=eb3bb297eaba44b4a706d4f7b0ae3550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.465570 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3601449ddebd4a57a460557971a87ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.465796 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92551c13d9dd4408a3609df997184b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.465963 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93fff736d93b4d638e6492aa1294df0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.466030 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet bc06a77162a6430bbbda7a3ca2c4e9ab
I20250115 14:52:17.466126 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd0c62e3692490ab5c24b0f6f430870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.466282 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66107249af824a0d87e0de1219f5b9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.470135 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3fd8b80954c40a0b7679e1dfa6b4671. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:17.470281 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 96692985a2c240099dbcb191fbd22384 consensus state has no leader
I20250115 14:52:17.470625 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3f53b19c7d544a56b0b4307eff6da294"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.471114 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f53b19c7d544a56b0b4307eff6da294 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.471632 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bccb7b153f04a6d87edec011d878c0a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:17.472213 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33193ebbb6e34f1785a1687f78ff1258 consensus state has no leader
I20250115 14:52:17.472417 448 master_service.cc:946] ReplaceTablet: received request to replace tablet e5dfc00ecec54f92a75fa77c84fe8a6d from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:17.472798 448 master_service.cc:946] ReplaceTablet: received request to replace tablet df0d072330444439a6b9b87db099c8b1 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:17.474364 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368de0e78e334d82a588a9ed89dc4b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.474496 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c672b93188548778a827cc745fe5aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.475353 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7de7f2b01a3474cb5bed484d8d9b01e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.475484 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69ef11c26b91409093b1322e5d426208. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.475790 2915 raft_consensus.cc:491] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.475914 2915 raft_consensus.cc:513] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.476523 2915 leader_election.cc:290] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:17.476562 746 tablet_service.cc:1467] Processing CreateTablet for tablet a3f636c8d2174702ab628551268724f5 (DEFAULT_TABLE table=default.Table_e8d3a3b353b14d8bb1f1c0afb200b824 [id=dcc82589e228441e8aff9ebb6fe5fff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.476856 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3f636c8d2174702ab628551268724f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.477191 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cb8dfa7c42480bbd139d5acf059a57" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:17.477684 983 leader_election.cc:343] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.480651 2549 raft_consensus.cc:491] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250115 14:52:17.480705 848 leader_election.cc:343] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.480808 848 leader_election.cc:304] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.480791 2549 raft_consensus.cc:513] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.481350 2549 leader_election.cc:290] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:17.481380 2835 raft_consensus.cc:2743] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:17.482343 847 leader_election.cc:343] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: ded663ac1197454bbde3c4c2d526002d
I20250115 14:52:17.484778 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6cb8dfa7c42480bbd139d5acf059a57" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.490304 452 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 4a79f880415a4b16aabdbb7cd989efdb (table default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a]). Expected version 2 got 0
I20250115 14:52:17.490509 452 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 246720dbb4ed4216ac87fb4a3df79e2c (table default.Table_daaa2755fe9d4f54ae6ff33ae452ec05 [id=8c0229e6f6884ea7be3417678bb27377]). Expected version 1 got 0
W20250115 14:52:17.494714 984 leader_election.cc:343] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.494840 984 leader_election.cc:304] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.495195 2915 raft_consensus.cc:2743] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.500787 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded663ac1197454bbde3c4c2d526002d" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.503244 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 417c5441bb7147a98b8ce0cd823a1855
I20250115 14:52:17.503657 448 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet df0d072330444439a6b9b87db099c8b1
I20250115 14:52:17.503885 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 87b28e83081a4f6e959c8b26c68b81ce
I20250115 14:52:17.504005 448 catalog_manager.cc:6596] ReplaceTablet: tablet df0d072330444439a6b9b87db099c8b1 deleted and replaced by tablet acc43b7b402249339f6bf213e35e35b9
I20250115 14:52:17.504724 751 tablet_service.cc:1514] Processing DeleteTablet for tablet bc06a77162a6430bbbda7a3ca2c4e9ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.507896 2915 raft_consensus.cc:491] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.508028 2915 raft_consensus.cc:513] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.508625 2915 leader_election.cc:290] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
W20250115 14:52:17.509003 848 leader_election.cc:343] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.509183 848 leader_election.cc:304] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.509159 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f3f6de654d4765b1d7a381f93b0c43" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:17.509475 2835 raft_consensus.cc:2743] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:17.509666 983 leader_election.cc:343] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.510596 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ad988e25800f4af9a069b5a80582f749
I20250115 14:52:17.510630 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8bae9867d745afb1dfdf47bcb881da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.510826 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0f3f6de654d4765b1d7a381f93b0c43" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:17.511224 984 leader_election.cc:343] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.511322 984 leader_election.cc:304] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.511593 889 tablet_service.cc:1467] Processing CreateTablet for tablet 92551c13d9dd4408a3609df997184b69 (DEFAULT_TABLE table=default.Table_35647648e7e64b1382583fb35ae12dc9 [id=9a3309da08bd4a309286d14ce1fb8ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.511824 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3bccb7b153f04a6d87edec011d878c0a (DEFAULT_TABLE table=default.Table_a2dac92e95de48bfb2af2bdd00b50c33 [id=4e4c188c02d443af99e85a602baa316c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.511950 877 tablet_service.cc:1467] Processing CreateTablet for tablet 9c672b93188548778a827cc745fe5aec (DEFAULT_TABLE table=default.Table_aac511b008a343a4a9b676f06ef31a5b [id=642161d0d6a0434f8c78aacf985c1eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.511974 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 9c672b93188548778a827cc745fe5aec (DEFAULT_TABLE table=default.Table_aac511b008a343a4a9b676f06ef31a5b [id=642161d0d6a0434f8c78aacf985c1eb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.512279 2915 raft_consensus.cc:2743] T d0f3f6de654d4765b1d7a381f93b0c43 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.512310 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c672b93188548778a827cc745fe5aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.512853 883 tablet_service.cc:1467] Processing CreateTablet for tablet e3fd8b80954c40a0b7679e1dfa6b4671 (DEFAULT_TABLE table=default.Table_75c1d7f0fd574f53a5b1a2853de03c30 [id=eb3bb297eaba44b4a706d4f7b0ae3550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.513152 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd0c62e3692490ab5c24b0f6f430870 (DEFAULT_TABLE table=default.Table_97011305bbd548d9bc1d1e2ea2730781 [id=04a97532b94a4b41a77418ca1861ac5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.513278 884 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd0c62e3692490ab5c24b0f6f430870 (DEFAULT_TABLE table=default.Table_97011305bbd548d9bc1d1e2ea2730781 [id=04a97532b94a4b41a77418ca1861ac5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.513453 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd0c62e3692490ab5c24b0f6f430870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.518337 746 tablet_service.cc:1514] Processing DeleteTablet for tablet 417c5441bb7147a98b8ce0cd823a1855 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.519977 746 tablet_service.cc:1514] Processing DeleteTablet for tablet ad988e25800f4af9a069b5a80582f749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.519977 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b28e83081a4f6e959c8b26c68b81ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.520984 1166 tablet_replica.cc:331] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:17.521139 751 tablet_service.cc:1514] Processing DeleteTablet for tablet df0d072330444439a6b9b87db099c8b1 with delete_type TABLET_DATA_DELETED (replaced by tablet acc43b7b402249339f6bf213e35e35b9) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.521167 1166 raft_consensus.cc:2238] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.521368 1166 raft_consensus.cc:2267] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.522084 1166 ts_tablet_manager.cc:1905] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.523677 2835 raft_consensus.cc:491] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.523794 2835 raft_consensus.cc:513] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.524423 2835 leader_election.cc:290] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:17.525408 847 leader_election.cc:343] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 48b15de9bf8943cdb9e5abd943484993
I20250115 14:52:17.525888 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48b15de9bf8943cdb9e5abd943484993" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:17.526551 848 leader_election.cc:343] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.526652 848 leader_election.cc:304] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.527024 2835 raft_consensus.cc:2743] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.527091 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c672b93188548778a827cc745fe5aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.527241 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3fd8b80954c40a0b7679e1dfa6b4671. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.527621 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bccb7b153f04a6d87edec011d878c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.528280 1166 ts_tablet_manager.cc:1918] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.528439 1166 log.cc:1198] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/8f8bae9867d745afb1dfdf47bcb881da
I20250115 14:52:17.528784 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92551c13d9dd4408a3609df997184b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.528789 1166 ts_tablet_manager.cc:1939] T 8f8bae9867d745afb1dfdf47bcb881da P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:17.530036 1166 tablet_replica.cc:331] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:17.530244 1166 raft_consensus.cc:2238] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.530436 1166 raft_consensus.cc:2267] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.530962 1166 ts_tablet_manager.cc:1905] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.531172 2915 raft_consensus.cc:491] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.531307 2915 raft_consensus.cc:513] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.531857 2915 leader_election.cc:290] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:17.532747 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a8cd2e6f3a465181ec781cc92bc2f6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.533120 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd0c62e3692490ab5c24b0f6f430870. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.533074 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9a8cd2e6f3a465181ec781cc92bc2f6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:17.533599 984 leader_election.cc:343] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:17.533807 983 leader_election.cc:343] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.533887 983 leader_election.cc:304] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.534400 2839 raft_consensus.cc:2743] T a9a8cd2e6f3a465181ec781cc92bc2f6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.536584 1013 tablet_service.cc:1467] Processing CreateTablet for tablet e3fd8b80954c40a0b7679e1dfa6b4671 (DEFAULT_TABLE table=default.Table_75c1d7f0fd574f53a5b1a2853de03c30 [id=eb3bb297eaba44b4a706d4f7b0ae3550]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.536991 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3fd8b80954c40a0b7679e1dfa6b4671. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.540570 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 92551c13d9dd4408a3609df997184b69 (DEFAULT_TABLE table=default.Table_35647648e7e64b1382583fb35ae12dc9 [id=9a3309da08bd4a309286d14ce1fb8ed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.540956 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92551c13d9dd4408a3609df997184b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.544598 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 3bccb7b153f04a6d87edec011d878c0a (DEFAULT_TABLE table=default.Table_a2dac92e95de48bfb2af2bdd00b50c33 [id=4e4c188c02d443af99e85a602baa316c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.545009 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bccb7b153f04a6d87edec011d878c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.548852 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet bc06a77162a6430bbbda7a3ca2c4e9ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.548918 885 tablet_service.cc:1514] Processing DeleteTablet for tablet bc06a77162a6430bbbda7a3ca2c4e9ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.548955 880 tablet_service.cc:1467] Processing CreateTablet for tablet 368de0e78e334d82a588a9ed89dc4b27 (DEFAULT_TABLE table=default.Table_5beff98980534f9694b89e4fa022ab17 [id=65192124c9a64955af16569fd4320422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.549249 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368de0e78e334d82a588a9ed89dc4b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.552944 1162 tablet_bootstrap.cc:492] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:17.553073 1162 ts_tablet_manager.cc:1397] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.120s user 0.002s sys 0.000s
I20250115 14:52:17.553977 1162 raft_consensus.cc:357] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.554185 1162 raft_consensus.cc:383] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.554234 1162 raft_consensus.cc:738] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.554407 1162 consensus_queue.cc:260] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.558715 2915 raft_consensus.cc:491] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.558835 2915 raft_consensus.cc:513] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.559362 2915 leader_election.cc:290] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:17.559701 1162 ts_tablet_manager.cc:1428] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:52:17.560110 1162 tablet_bootstrap.cc:492] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:17.561379 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 417c5441bb7147a98b8ce0cd823a1855 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.561400 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 417c5441bb7147a98b8ce0cd823a1855 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.561342 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57aecd22f97c41618f90d14c2f53a7c9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:17.561995 983 leader_election.cc:343] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.562103 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet df0d072330444439a6b9b87db099c8b1 with delete_type TABLET_DATA_DELETED (replaced by tablet acc43b7b402249339f6bf213e35e35b9) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.562136 885 tablet_service.cc:1514] Processing DeleteTablet for tablet df0d072330444439a6b9b87db099c8b1 with delete_type TABLET_DATA_DELETED (replaced by tablet acc43b7b402249339f6bf213e35e35b9) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.562681 885 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b28e83081a4f6e959c8b26c68b81ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.562726 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 87b28e83081a4f6e959c8b26c68b81ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.563279 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 6203efb862ea4962a6d09f3e4306a92f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.563284 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57aecd22f97c41618f90d14c2f53a7c9" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:17.563827 984 leader_election.cc:343] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.563925 984 leader_election.cc:304] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.564343 2915 raft_consensus.cc:2743] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.564646 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet ad988e25800f4af9a069b5a80582f749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.565549 1162 tablet_bootstrap.cc:654] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:17.562448 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet bc06a77162a6430bbbda7a3ca2c4e9ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc06a77162a6430bbbda7a3ca2c4e9ab already in progress: creating tablet
I20250115 14:52:17.566525 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 8f8bae9867d745afb1dfdf47bcb881da (table default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]) successfully deleted
W20250115 14:52:17.566982 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 417c5441bb7147a98b8ce0cd823a1855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417c5441bb7147a98b8ce0cd823a1855 already in progress: creating tablet
I20250115 14:52:17.567219 1168 tablet_bootstrap.cc:492] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:17.569264 748 tablet_service.cc:1467] Processing CreateTablet for tablet 9948d4a0552a4dfcad497d4fe1472fb1 (DEFAULT_TABLE table=default.Table_4fe9c35efdec44e1b24937fe275c2b3c [id=c64e70cd3ca64677bf45dc26fc2c6d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.569453 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_78ff869a9d0c483a87b4d43f8bcac7c9" } modify_external_catalogs: true
I20250115 14:52:17.569625 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_78ff869a9d0c483a87b4d43f8bcac7c9" } modify_external_catalogs: true
I20250115 14:52:17.569681 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9948d4a0552a4dfcad497d4fe1472fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.571830 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b3cca34664f749cd9a8646474858530b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.572351 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3cca34664f749cd9a8646474858530b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.573030 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cc2f5585e4594d849b5c784d7e6882cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.573446 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc2f5585e4594d849b5c784d7e6882cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.573689 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f4d5908220b548f1ba60227f435a948a" } new_table_name: "default.Table_61c9adac2720423b8be3226a806f6547" modify_external_catalogs: true
I20250115 14:52:17.577181 880 tablet_service.cc:1514] Processing DeleteTablet for tablet ad988e25800f4af9a069b5a80582f749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.577382 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 368de0e78e334d82a588a9ed89dc4b27 (DEFAULT_TABLE table=default.Table_5beff98980534f9694b89e4fa022ab17 [id=65192124c9a64955af16569fd4320422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.577751 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 368de0e78e334d82a588a9ed89dc4b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.586354 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 9948d4a0552a4dfcad497d4fe1472fb1 (DEFAULT_TABLE table=default.Table_4fe9c35efdec44e1b24937fe275c2b3c [id=c64e70cd3ca64677bf45dc26fc2c6d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.586438 880 tablet_service.cc:1467] Processing CreateTablet for tablet 9948d4a0552a4dfcad497d4fe1472fb1 (DEFAULT_TABLE table=default.Table_4fe9c35efdec44e1b24937fe275c2b3c [id=c64e70cd3ca64677bf45dc26fc2c6d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.586794 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9948d4a0552a4dfcad497d4fe1472fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.586807 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9948d4a0552a4dfcad497d4fe1472fb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.590900 1162 tablet_bootstrap.cc:492] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:17.591200 1162 ts_tablet_manager.cc:1397] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250115 14:52:17.596107 1046 raft_consensus.cc:1270] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Refusing update from remote peer 73c86158cc2641f182ea50a06634592d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:17.596880 2835 consensus_queue.cc:1035] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250115 14:52:17.598568 848 consensus_peers.cc:487] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:17.606009 908 raft_consensus.cc:1270] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Refusing update from remote peer c238c037a1c74725a001e2b4a1ff4304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250115 14:52:17.606627 2915 consensus_queue.cc:1035] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250115 14:52:17.607004 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_06fdf72400894754b2da4a05ae517fac" } new_table_name: "default.Table_40f4c6338a424d53aabbab3caf5b9843" modify_external_catalogs: true
I20250115 14:52:17.609578 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_213f670682f841518bdfae30c8c19d67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.610164 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_213f670682f841518bdfae30c8c19d67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.607107 1162 raft_consensus.cc:357] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.611454 1162 raft_consensus.cc:383] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.611939 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7052ed5e2bf04d48b7d75dd702094b65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:17.612179 1162 raft_consensus.cc:738] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
W20250115 14:52:17.614171 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7052ed5e2bf04d48b7d75dd702094b65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.614737 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_9c6fe96ae2fd4ef790e39ed73291a0bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.615199 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c6fe96ae2fd4ef790e39ed73291a0bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.619774 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9449e1d09c3148ee86074a57c8e22240
I20250115 14:52:17.620687 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7718729182a340fe9b2dfea5a8321c1f (table default.Table_b2803f567cf849a480c49e93808b1a19 [id=199476de8a584f9e9f6bcf826e83146b]). Expected version 2 got 0
I20250115 14:52:17.621603 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 9449e1d09c3148ee86074a57c8e22240 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.621599 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 9449e1d09c3148ee86074a57c8e22240 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.536243 1166 ts_tablet_manager.cc:1918] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:17.621841 1166 log.cc:1198] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ad988e25800f4af9a069b5a80582f749
I20250115 14:52:17.622516 1166 ts_tablet_manager.cc:1939] T ad988e25800f4af9a069b5a80582f749 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:17.624241 1162 consensus_queue.cc:260] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.625619 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1326) took 1491 ms. Trace:
I20250115 14:52:17.625707 1167 rpcz_store.cc:276] 0115 14:52:16.134089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.134301 (+ 212us) service_pool.cc:225] Handling call
0115 14:52:17.112350 (+978049us) ts_tablet_manager.cc:1145] Deleting tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6
0115 14:52:17.112360 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.139771 (+ 27411us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.208038 (+ 68267us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.625603 (+417565us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":977861}
I20250115 14:52:17.626089 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 3c2ee5dfdef14ec092a2f9d4bfd295a6 (table default.Table_ebfbdc698c8c448ca2e60757b1fd574c [id=471b029f1dba4dd087deb339bba02013]) successfully deleted
I20250115 14:52:17.629482 2774 tablet.cc:1721] T 09adb34bf9a3464798b9f28c50d6fc11 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:17.630741 2908 tablet.cc:1721] T 09adb34bf9a3464798b9f28c50d6fc11 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:17.631258 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2988e8cd3886441db242a362b06544c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.631729 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2988e8cd3886441db242a362b06544c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.634546 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9449e1d09c3148ee86074a57c8e22240 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.635193 1162 ts_tablet_manager.cc:1428] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.044s user 0.003s sys 0.000s
I20250115 14:52:17.635393 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ad988e25800f4af9a069b5a80582f749 (table default.Table_162eb94aa1674d21ae644021401a54e8 [id=e3887aa962874cfaa7ee89503ca1a49a]) successfully deleted
W20250115 14:52:17.635596 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 87b28e83081a4f6e959c8b26c68b81ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87b28e83081a4f6e959c8b26c68b81ce already in progress: creating tablet
W20250115 14:52:17.635740 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet df0d072330444439a6b9b87db099c8b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df0d072330444439a6b9b87db099c8b1 already in progress: creating tablet
W20250115 14:52:17.635881 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 9449e1d09c3148ee86074a57c8e22240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9449e1d09c3148ee86074a57c8e22240 already in progress: creating tablet
I20250115 14:52:17.636749 1162 tablet_bootstrap.cc:492] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:17.638044 1162 tablet_bootstrap.cc:654] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.639021 1167 tablet_replica.cc:331] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:17.639222 1167 raft_consensus.cc:2238] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.642546 1167 raft_consensus.cc:2267] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.650173 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 7718729182a340fe9b2dfea5a8321c1f (table default.Table_b2803f567cf849a480c49e93808b1a19 [id=199476de8a584f9e9f6bcf826e83146b]). Expected version 2 got 0
I20250115 14:52:17.650410 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet e49dc97eea474252a5e0ef4af6e015a9 (table default.Table_4f00ef2ef63d4f77929a245aa9bff035 [id=8502b5fe264f4fb5a89bd4a91949c90b]). Expected version 1 got 0
I20250115 14:52:17.652609 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fddbd9ba14443a91865387e1d1c238 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.655897 2849 tablet.cc:1721] T e49dc97eea474252a5e0ef4af6e015a9 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.661120 2774 tablet.cc:1721] T e49dc97eea474252a5e0ef4af6e015a9 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.663070 1167 ts_tablet_manager.cc:1905] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.666559 1751 ts_tablet_manager.cc:1918] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.666776 1751 log.cc:1198] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/cc9ce226d3ba44d6908257bbd0df3a12
I20250115 14:52:17.667161 1751 ts_tablet_manager.cc:1939] T cc9ce226d3ba44d6908257bbd0df3a12 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:17.668219 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 068771fee92f403d8997646291a83dc6 consensus state has no leader
W20250115 14:52:17.668709 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c487e956367845299e20cea05ce52987 consensus state has no leader
I20250115 14:52:17.670171 449 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_2e56658df72047609417d02e32f69b18 [id=37297c9e757d411daac43cbbea813e0c] alter complete (version 2)
I20250115 14:52:17.671003 1164 tablet_bootstrap.cc:492] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.671196 1164 ts_tablet_manager.cc:1397] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.352s user 0.007s sys 0.016s
I20250115 14:52:17.672178 1164 raft_consensus.cc:357] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.672456 1164 raft_consensus.cc:383] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.672544 1164 raft_consensus.cc:738] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.672796 1164 consensus_queue.cc:260] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.673354 1164 ts_tablet_manager.cc:1428] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:17.673915 1164 tablet_bootstrap.cc:492] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:17.675279 1164 tablet_bootstrap.cc:654] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.681001 1162 tablet_bootstrap.cc:492] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:17.681142 1162 ts_tablet_manager.cc:1397] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20250115 14:52:17.681492 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ded663ac1197454bbde3c4c2d526002d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.681788 1166 tablet_replica.cc:331] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:17.681735 1162 raft_consensus.cc:357] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.681906 1162 raft_consensus.cc:383] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.681946 1162 raft_consensus.cc:738] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.681957 1166 raft_consensus.cc:2238] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.682092 1166 raft_consensus.cc:2267] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.682099 1162 consensus_queue.cc:260] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.682926 1166 ts_tablet_manager.cc:1905] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.682502 1162 ts_tablet_manager.cc:1428] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250115 14:52:17.688565 1162 tablet_bootstrap.cc:492] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:17.688830 1166 ts_tablet_manager.cc:1918] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.688992 1166 log.cc:1198] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ded663ac1197454bbde3c4c2d526002d
I20250115 14:52:17.689318 1166 ts_tablet_manager.cc:1939] T ded663ac1197454bbde3c4c2d526002d P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:17.690136 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4b828508be354eb4a8f42f5ba4b2e760" } new_table_name: "default.Table_7a0fe9ccb59e4b3486ecae42f1564438" modify_external_catalogs: true
I20250115 14:52:17.692076 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ded663ac1197454bbde3c4c2d526002d (table default.Table_ca5ad5415b9144e583b4c771d24d4431 [id=a9793eda46fd4d07963603331438decb]) successfully deleted
I20250115 14:52:17.692590 452 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_28d063764bb94657a5f118c8820e8fbe" } modify_external_catalogs: true
I20250115 14:52:17.692741 452 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_28d063764bb94657a5f118c8820e8fbe" } modify_external_catalogs: true
I20250115 14:52:17.693215 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_adcc45a64d8840fba4e22623a802abb6" } new_table_name: "default.Table_c3f7a19b18394aea86b01a3a6d969d87" modify_external_catalogs: true
I20250115 14:52:17.694303 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d01041829a3b4c529cf198b45e9a0d3f" } new_table_name: "default.Table_4a28b224fc5b4eea8c814891c8b5188e" modify_external_catalogs: true
I20250115 14:52:17.695065 1162 tablet_bootstrap.cc:654] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.695441 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3ba14975b9434368bb1ad7ce335a7baa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.695925 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ba14975b9434368bb1ad7ce335a7baa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.697069 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_896e88aa54b34b63a9d7eb65d358e71b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.697449 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_896e88aa54b34b63a9d7eb65d358e71b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.699263 1164 tablet_bootstrap.cc:492] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.699434 1164 ts_tablet_manager.cc:1397] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.004s
I20250115 14:52:17.699529 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1342) took 1165 ms. Trace:
I20250115 14:52:17.699750 1751 rpcz_store.cc:276] 0115 14:52:16.534185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.534228 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:17.429675 (+895447us) ts_tablet_manager.cc:1145] Deleting tablet cc9ce226d3ba44d6908257bbd0df3a12
0115 14:52:17.429684 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.451283 (+ 21599us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.666419 (+215136us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.699509 (+ 33090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":895303}
I20250115 14:52:17.700202 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet cc9ce226d3ba44d6908257bbd0df3a12 (table default.Table_57e02beccdd64ec6ad08a1f3d0908b65 [id=ceccf8508c5047c2b7a63788dd81a66f]) successfully deleted
I20250115 14:52:17.700601 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1339) took 1176 ms. Trace:
I20250115 14:52:17.700399 1164 raft_consensus.cc:357] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.700673 1751 rpcz_store.cc:276] 0115 14:52:16.524203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.524255 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:17.700479 (+1176224us) ts_tablet_manager.cc:1145] Deleting tablet d27a88650b024af19938618d9ed0cc23
0115 14:52:17.700489 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.700585 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1162945}
I20250115 14:52:17.700963 1164 raft_consensus.cc:383] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250115 14:52:17.701150 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet d27a88650b024af19938618d9ed0cc23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d27a88650b024af19938618d9ed0cc23 already in progress: creating tablet
I20250115 14:52:17.701210 1164 raft_consensus.cc:738] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.701481 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1340) took 1172 ms. Trace:
I20250115 14:52:17.701556 1751 rpcz_store.cc:276] 0115 14:52:16.528637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.528702 (+ 65us) service_pool.cc:225] Handling call
0115 14:52:17.701379 (+1172677us) ts_tablet_manager.cc:1145] Deleting tablet afea664025e54f6099d825c8f7a0ea1a
0115 14:52:17.701388 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.701467 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1163808}
W20250115 14:52:17.701892 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet afea664025e54f6099d825c8f7a0ea1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afea664025e54f6099d825c8f7a0ea1a already in progress: creating tablet
I20250115 14:52:17.701534 1164 consensus_queue.cc:260] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.703274 1164 ts_tablet_manager.cc:1428] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:17.703460 1167 ts_tablet_manager.cc:1918] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.703567 1167 log.cc:1198] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/8f8bae9867d745afb1dfdf47bcb881da
I20250115 14:52:17.703838 1167 ts_tablet_manager.cc:1939] T 8f8bae9867d745afb1dfdf47bcb881da P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:17.704097 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1323) took 1570 ms. Trace:
I20250115 14:52:17.704167 1167 rpcz_store.cc:276] 0115 14:52:16.133753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.134411 (+ 658us) service_pool.cc:225] Handling call
0115 14:52:17.638970 (+1504559us) ts_tablet_manager.cc:1145] Deleting tablet 8f8bae9867d745afb1dfdf47bcb881da
0115 14:52:17.638981 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.689230 (+ 50249us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.703355 (+ 14125us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:17.704085 (+ 730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1504424}
I20250115 14:52:17.704739 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 8f8bae9867d745afb1dfdf47bcb881da (table default.Table_7b4c900318f642499ecffcb268290262 [id=181260f8ab5643d4a541a21bdde9b7b3]) successfully deleted
I20250115 14:52:17.704794 1164 tablet_bootstrap.cc:492] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:17.707732 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet e49dc97eea474252a5e0ef4af6e015a9 (table default.Table_4f00ef2ef63d4f77929a245aa9bff035 [id=8502b5fe264f4fb5a89bd4a91949c90b]). Expected version 1 got 0
I20250115 14:52:17.708693 1164 tablet_bootstrap.cc:654] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.709586 455 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_4f00ef2ef63d4f77929a245aa9bff035 [id=8502b5fe264f4fb5a89bd4a91949c90b] alter complete (version 1)
I20250115 14:52:17.714099 1168 tablet_bootstrap.cc:654] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.716813 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 8edddd03b4f34de4bb4801a6dcca183b (table default.Table_aacabd4362854aeaa394ec57852c2ccc [id=5c127085c24a42688f4d0f2808d6ffc7]). Expected version 1 got 0
I20250115 14:52:17.719237 446 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3089c6f63edb4642b0ed0a82ee1b5051" } modify_external_catalogs: true
I20250115 14:52:17.719396 446 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3089c6f63edb4642b0ed0a82ee1b5051" } modify_external_catalogs: true
I20250115 14:52:17.719561 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7d66aaaa255c4c1a9672e2dd8b26a18e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:17.719800 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f0e1896dc4ac463398d14176e3deda35" } modify_external_catalogs: true
I20250115 14:52:17.719910 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f0e1896dc4ac463398d14176e3deda35" } modify_external_catalogs: true
W20250115 14:52:17.720052 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d66aaaa255c4c1a9672e2dd8b26a18e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.721079 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f586057adf44494596993a543597c699"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.721483 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f586057adf44494596993a543597c699 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.722004 2915 raft_consensus.cc:491] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.722159 2915 raft_consensus.cc:513] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.722765 2915 leader_election.cc:290] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:17.724484 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342bf8d4dc4244f982b1b63aaae0fb45" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:17.725284 983 leader_election.cc:343] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.725893 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342bf8d4dc4244f982b1b63aaae0fb45" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:17.726462 984 leader_election.cc:343] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.726609 984 leader_election.cc:304] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.726864 2915 raft_consensus.cc:2743] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.718374 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1327) took 1584 ms. Trace:
I20250115 14:52:17.727947 1167 rpcz_store.cc:276] 0115 14:52:16.134170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.134247 (+ 77us) service_pool.cc:225] Handling call
0115 14:52:17.718249 (+1584002us) ts_tablet_manager.cc:1145] Deleting tablet 87026b6faf75437eac21dedab1f41ed4
0115 14:52:17.718260 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.718356 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1582782}
I20250115 14:52:17.728468 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1328) took 1374 ms. Trace:
I20250115 14:52:17.728538 1167 rpcz_store.cc:276] 0115 14:52:16.353826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.356920 (+ 3094us) service_pool.cc:225] Handling call
0115 14:52:17.728356 (+1371436us) ts_tablet_manager.cc:1145] Deleting tablet b3daf6416ad847aeaa83087cc81c9239
0115 14:52:17.728367 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.728453 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1339922}
W20250115 14:52:17.729034 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 87026b6faf75437eac21dedab1f41ed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87026b6faf75437eac21dedab1f41ed4 already in progress: creating tablet
W20250115 14:52:17.729290 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b3daf6416ad847aeaa83087cc81c9239 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3daf6416ad847aeaa83087cc81c9239 already in progress: creating tablet
W20250115 14:52:17.729717 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1a543188b2ca4478b79d4ad08ee30b89 consensus state has no leader
I20250115 14:52:17.729875 1164 tablet_bootstrap.cc:492] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.730304 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_06ae3b8ba32f4b05af77d455a746ecd2" } modify_external_catalogs: true
I20250115 14:52:17.730413 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1329) took 1329 ms. Trace:
I20250115 14:52:17.730492 1167 rpcz_store.cc:276] 0115 14:52:16.401248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.401293 (+ 45us) service_pool.cc:225] Handling call
0115 14:52:17.730313 (+1329020us) ts_tablet_manager.cc:1145] Deleting tablet 7c3393ed28d242838d6f2f170184f7e8
0115 14:52:17.730323 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.730397 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1328752}
I20250115 14:52:17.730532 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_06ae3b8ba32f4b05af77d455a746ecd2" } modify_external_catalogs: true
I20250115 14:52:17.730883 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1330) took 1329 ms. Trace:
I20250115 14:52:17.730890 1168 tablet_bootstrap.cc:492] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:17.730963 1167 rpcz_store.cc:276] 0115 14:52:16.401688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.401724 (+ 36us) service_pool.cc:225] Handling call
0115 14:52:17.730790 (+1329066us) ts_tablet_manager.cc:1145] Deleting tablet 4968ce56c6b4457097880a8ea7f924d8
0115 14:52:17.730798 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.730869 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1328878}
W20250115 14:52:17.731114 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7c3393ed28d242838d6f2f170184f7e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c3393ed28d242838d6f2f170184f7e8 already in progress: creating tablet
I20250115 14:52:17.731045 1168 ts_tablet_manager.cc:1397] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.164s user 0.002s sys 0.009s
W20250115 14:52:17.731374 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 4968ce56c6b4457097880a8ea7f924d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4968ce56c6b4457097880a8ea7f924d8 already in progress: creating tablet
I20250115 14:52:17.731953 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1331) took 1329 ms. Trace:
I20250115 14:52:17.730005 1164 ts_tablet_manager.cc:1397] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20250115 14:52:17.732020 1167 rpcz_store.cc:276] 0115 14:52:16.402504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.402572 (+ 68us) service_pool.cc:225] Handling call
0115 14:52:17.731852 (+1329280us) ts_tablet_manager.cc:1145] Deleting tablet 7db35ff0383549de86230bb1eb89389f
0115 14:52:17.731861 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.731936 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1329085}
W20250115 14:52:17.733914 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7db35ff0383549de86230bb1eb89389f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7db35ff0383549de86230bb1eb89389f already in progress: creating tablet
W20250115 14:52:17.734155 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3f99d417ef074f978a91236d31552315 consensus state has no leader
W20250115 14:52:17.733744 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c519f995c4c413c85be48c4577d7db2 consensus state has no leader
I20250115 14:52:17.735605 1162 tablet_bootstrap.cc:492] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:17.735774 1162 ts_tablet_manager.cc:1397] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.001s
I20250115 14:52:17.736096 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1332) took 1211 ms. Trace:
I20250115 14:52:17.736173 1167 rpcz_store.cc:276] 0115 14:52:16.524170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.524224 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:17.735982 (+1211758us) ts_tablet_manager.cc:1145] Deleting tablet d27a88650b024af19938618d9ed0cc23
0115 14:52:17.735991 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.736080 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1211556}
W20250115 14:52:17.736707 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet d27a88650b024af19938618d9ed0cc23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d27a88650b024af19938618d9ed0cc23 already in progress: creating tablet
I20250115 14:52:17.736788 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1333) took 1208 ms. Trace:
I20250115 14:52:17.736771 1162 raft_consensus.cc:357] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.736852 1167 rpcz_store.cc:276] 0115 14:52:16.528564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.528607 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:17.736700 (+1208093us) ts_tablet_manager.cc:1145] Deleting tablet afea664025e54f6099d825c8f7a0ea1a
0115 14:52:17.736708 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.736775 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1207909}
I20250115 14:52:17.736977 1162 raft_consensus.cc:383] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.737033 1162 raft_consensus.cc:738] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
W20250115 14:52:17.737177 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet afea664025e54f6099d825c8f7a0ea1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afea664025e54f6099d825c8f7a0ea1a already in progress: creating tablet
I20250115 14:52:17.737272 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1334) took 1171 ms. Trace:
I20250115 14:52:17.737222 1162 consensus_queue.cc:260] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.737329 1167 rpcz_store.cc:276] 0115 14:52:16.566239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.566291 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:17.737184 (+1170893us) ts_tablet_manager.cc:1145] Deleting tablet ffa9cd1279fe4e0c9168472993a983dd
0115 14:52:17.737193 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.737260 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1170702}
W20250115 14:52:17.737761 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ffa9cd1279fe4e0c9168472993a983dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffa9cd1279fe4e0c9168472993a983dd already in progress: creating tablet
I20250115 14:52:17.732038 1168 raft_consensus.cc:357] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.738168 1168 raft_consensus.cc:383] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.738231 1168 raft_consensus.cc:738] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.738420 1168 consensus_queue.cc:260] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.738926 1168 ts_tablet_manager.cc:1428] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250115 14:52:17.739252 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1343) took 1173 ms. Trace:
I20250115 14:52:17.739370 1751 rpcz_store.cc:276] 0115 14:52:16.566309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:16.566432 (+ 123us) service_pool.cc:225] Handling call
0115 14:52:17.739133 (+1172701us) ts_tablet_manager.cc:1145] Deleting tablet ffa9cd1279fe4e0c9168472993a983dd
0115 14:52:17.739143 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:17.739235 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172551}
I20250115 14:52:17.739781 1164 raft_consensus.cc:357] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.739971 1164 raft_consensus.cc:383] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.740006 1162 ts_tablet_manager.cc:1428] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250115 14:52:17.740026 1164 raft_consensus.cc:738] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.740267 1167 tablet_replica.cc:331] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
W20250115 14:52:17.740272 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet ffa9cd1279fe4e0c9168472993a983dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffa9cd1279fe4e0c9168472993a983dd already in progress: creating tablet
I20250115 14:52:17.740685 1168 tablet_bootstrap.cc:492] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:17.742326 1164 consensus_queue.cc:260] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.742926 1164 ts_tablet_manager.cc:1428] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250115 14:52:17.742926 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d612b424bea408cb6691c7c200cf3a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.743194 1162 tablet_bootstrap.cc:492] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:17.743182 1164 tablet_bootstrap.cc:492] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:17.741315 1167 raft_consensus.cc:2238] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:17.744415 1167 pending_rounds.cc:62] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: Trying to abort 1 pending ops.
I20250115 14:52:17.744472 1167 pending_rounds.cc:69] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114558408862031872 op_type: NO_OP noop_request { }
I20250115 14:52:17.744570 1167 raft_consensus.cc:2883] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250115 14:52:17.744645 1167 raft_consensus.cc:2267] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.745153 1167 ts_tablet_manager.cc:1905] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.746392 1164 tablet_bootstrap.cc:654] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.750406 1164 tablet_bootstrap.cc:492] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.750561 1164 ts_tablet_manager.cc:1397] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:52:17.751209 1164 raft_consensus.cc:357] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.751277 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet a5e97279185541b992340c24ede01656 (table default.Table_a8391687956e4259a71808f267cb4ff0 [id=361ed20953ab4e79b1112ae76b95a423]). Expected version 3 got 0
I20250115 14:52:17.751391 1164 raft_consensus.cc:383] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.751444 1164 raft_consensus.cc:738] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.742224 1168 tablet_bootstrap.cc:654] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.753522 1162 tablet_bootstrap.cc:654] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.753481 1164 consensus_queue.cc:260] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.754853 1164 ts_tablet_manager.cc:1428] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:17.755335 1164 tablet_bootstrap.cc:492] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:17.758044 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet b6fb388c17044f5eabce9996ebbf4ff6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6fb388c17044f5eabce9996ebbf4ff6 already in progress: creating tablet
W20250115 14:52:17.758749 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e619d4aabff245f4b46f5d03cb9a856d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e619d4aabff245f4b46f5d03cb9a856d already in progress: creating tablet
I20250115 14:52:17.758967 1164 tablet_bootstrap.cc:654] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:17.759382 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet de27f6c600b840c2b00a881c5ef038ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de27f6c600b840c2b00a881c5ef038ae already in progress: creating tablet
I20250115 14:52:17.759639 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 398ac8af619c427287b3964443173864 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.759860 1166 tablet_replica.cc:331] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
W20250115 14:52:17.759949 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 2955ecdd45364dc2934ba96a9a6c8e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2955ecdd45364dc2934ba96a9a6c8e8b already in progress: creating tablet
I20250115 14:52:17.760025 1166 raft_consensus.cc:2238] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.760157 1166 raft_consensus.cc:2267] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.760200 1751 tablet_replica.cc:331] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:17.760722 1166 ts_tablet_manager.cc:1905] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.765748 2849 tablet.cc:1721] T 09adb34bf9a3464798b9f28c50d6fc11 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:17.772012 1166 ts_tablet_manager.cc:1918] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.772135 1166 log.cc:1198] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/398ac8af619c427287b3964443173864
I20250115 14:52:17.772480 1166 ts_tablet_manager.cc:1939] T 398ac8af619c427287b3964443173864 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:17.773403 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 398ac8af619c427287b3964443173864 (table default.Table_a26e212086b1478cb159097102934dd1 [id=4bb94d3cca0c4a14aad6957911139e51]) successfully deleted
I20250115 14:52:17.778609 1167 ts_tablet_manager.cc:1918] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250115 14:52:17.778713 1167 log.cc:1198] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/86c04fa23e7c40a5b5e07852e5d74a27
I20250115 14:52:17.778877 1164 tablet_bootstrap.cc:492] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.779001 1164 ts_tablet_manager.cc:1397] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250115 14:52:17.779011 1167 ts_tablet_manager.cc:1939] T 86c04fa23e7c40a5b5e07852e5d74a27 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:17.780586 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 86c04fa23e7c40a5b5e07852e5d74a27 (table default.Table_702932472ce94b91b4535e0015a38be1 [id=04c521c944614e03be02fcb2ee8499c2]) successfully deleted
I20250115 14:52:17.780735 1167 tablet_replica.cc:331] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:17.780920 1167 raft_consensus.cc:2238] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.781070 1167 raft_consensus.cc:2267] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.781564 1167 ts_tablet_manager.cc:1905] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.781735 1162 tablet_bootstrap.cc:492] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:17.781874 1162 ts_tablet_manager.cc:1397] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.003s
I20250115 14:52:17.782138 1164 raft_consensus.cc:357] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.782327 1164 raft_consensus.cc:383] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.782384 1164 raft_consensus.cc:738] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.782598 1162 raft_consensus.cc:357] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.782811 1162 raft_consensus.cc:383] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.782867 1162 raft_consensus.cc:738] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.783059 1162 consensus_queue.cc:260] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.783272 1164 consensus_queue.cc:260] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.784976 1164 ts_tablet_manager.cc:1428] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250115 14:52:17.785327 2835 raft_consensus.cc:491] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.785512 1162 ts_tablet_manager.cc:1428] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250115 14:52:17.785445 2835 raft_consensus.cc:513] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.785740 1162 tablet_bootstrap.cc:492] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:17.786065 2835 leader_election.cc:290] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:17.786314 1164 tablet_bootstrap.cc:492] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:17.786851 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba33eb3f5a40450db5cf115fbe8a17e7" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:17.787341 848 leader_election.cc:343] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.788549 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet d18a88c8f3ce4a24b7e1e13966760364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:08 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:17.788763 847 leader_election.cc:343] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: ba33eb3f5a40450db5cf115fbe8a17e7
I20250115 14:52:17.788867 847 leader_election.cc:304] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.789213 2835 raft_consensus.cc:2743] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.789786 1164 tablet_bootstrap.cc:654] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.790028 1162 tablet_bootstrap.cc:654] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.801645 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 6308f68e41554470b7d1321c60d4eee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.801972 1166 tablet_replica.cc:331] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:17.802140 1166 raft_consensus.cc:2238] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.802275 1166 raft_consensus.cc:2267] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.803243 1166 ts_tablet_manager.cc:1905] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.806164 1167 ts_tablet_manager.cc:1918] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.806253 1167 log.cc:1198] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/2e4c2707915e4e97ad6c66cb249c6d9e
I20250115 14:52:17.806492 1167 ts_tablet_manager.cc:1939] T 2e4c2707915e4e97ad6c66cb249c6d9e P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:17.807036 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 2e4c2707915e4e97ad6c66cb249c6d9e (table default.Table_eb0e928b8265482990bb5fbf1072985f [id=dfcc6db7be294db4a0f620ab93714f2b]) successfully deleted
I20250115 14:52:17.808763 1751 raft_consensus.cc:2238] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.808938 1751 raft_consensus.cc:2267] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.809458 1751 ts_tablet_manager.cc:1905] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.811141 1166 ts_tablet_manager.cc:1918] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.811302 1166 log.cc:1198] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/6308f68e41554470b7d1321c60d4eee9
I20250115 14:52:17.811645 1166 ts_tablet_manager.cc:1939] T 6308f68e41554470b7d1321c60d4eee9 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:17.812238 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ddb66753bb00448c8030b0e0ff03bb22
I20250115 14:52:17.812907 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb66753bb00448c8030b0e0ff03bb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.813386 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb66753bb00448c8030b0e0ff03bb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:17.813578 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.2:44187, user_credentials={real_user=slave}} blocked reactor thread for 238738us
I20250115 14:52:17.814293 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 6308f68e41554470b7d1321c60d4eee9 (table default.Table_33f4705d7a1b474d82734fe4a80c0424 [id=b0ebc66f46364e51b73bdee697f00b62]) successfully deleted
I20250115 14:52:17.815411 880 tablet_service.cc:1514] Processing DeleteTablet for tablet ddb66753bb00448c8030b0e0ff03bb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:17.831301 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 2955ecdd45364dc2934ba96a9a6c8e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2955ecdd45364dc2934ba96a9a6c8e8b already in progress: creating tablet
W20250115 14:52:17.832305 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet b6fb388c17044f5eabce9996ebbf4ff6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6fb388c17044f5eabce9996ebbf4ff6 already in progress: creating tablet
W20250115 14:52:17.832554 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet de27f6c600b840c2b00a881c5ef038ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de27f6c600b840c2b00a881c5ef038ae already in progress: creating tablet
W20250115 14:52:17.832788 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e619d4aabff245f4b46f5d03cb9a856d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e619d4aabff245f4b46f5d03cb9a856d already in progress: creating tablet
W20250115 14:52:17.833003 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ddb66753bb00448c8030b0e0ff03bb22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddb66753bb00448c8030b0e0ff03bb22 already in progress: creating tablet
I20250115 14:52:17.834338 2774 tablet.cc:1721] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.834533 455 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 (table default.Table_a3ebaf4214574dd68c619382cb749409 [id=c66ea162404c4eb0a012128385984399]). Expected version 1 got 0
I20250115 14:52:17.836536 1167 tablet_replica.cc:331] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:17.837253 455 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e79474d461c04307bd06aac8b3a66328" } modify_external_catalogs: true
I20250115 14:52:17.837459 455 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e79474d461c04307bd06aac8b3a66328" } modify_external_catalogs: true
I20250115 14:52:17.836723 1167 raft_consensus.cc:2238] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.838330 1167 raft_consensus.cc:2267] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.838848 1167 ts_tablet_manager.cc:1905] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.842630 2839 raft_consensus.cc:491] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.842767 2839 raft_consensus.cc:513] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.843376 2839 leader_election.cc:290] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:17.843904 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edddd03b4f34de4bb4801a6dcca183b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:17.844316 2865 raft_consensus.cc:491] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.844468 2865 raft_consensus.cc:513] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.845081 2865 leader_election.cc:290] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:17.845458 983 leader_election.cc:343] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.845482 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:17.846189 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edddd03b4f34de4bb4801a6dcca183b" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.846678 880 tablet_service.cc:1514] Processing DeleteTablet for tablet ba33eb3f5a40450db5cf115fbe8a17e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.846931 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49dc97eea474252a5e0ef4af6e015a9" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
W20250115 14:52:17.847388 984 leader_election.cc:343] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.847384 710 leader_election.cc:304] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:17.847613 2865 raft_consensus.cc:2743] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.847533 984 leader_election.cc:304] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.848085 2839 raft_consensus.cc:2743] T 8edddd03b4f34de4bb4801a6dcca183b P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.850852 2865 raft_consensus.cc:491] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.850978 2865 raft_consensus.cc:513] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.851536 2865 leader_election.cc:290] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:17.852017 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e97279185541b992340c24ede01656" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:17.852605 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5e97279185541b992340c24ede01656" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:17.853183 710 leader_election.cc:304] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:17.853873 2865 raft_consensus.cc:2743] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.854369 2849 tablet.cc:1721] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:17.858500 453 catalog_manager.cc:5618] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187) does not have the latest schema for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 (table default.Table_a3ebaf4214574dd68c619382cb749409 [id=c66ea162404c4eb0a012128385984399]). Expected version 1 got 0
I20250115 14:52:17.869232 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_aabd2a7266164dffa263161d840451bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.869843 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aabd2a7266164dffa263161d840451bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.870105 1167 ts_tablet_manager.cc:1918] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.870198 1167 log.cc:1198] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/6e2d55a4e32448fbb98f704dbf1657b9
I20250115 14:52:17.871065 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_24972c0535bf41ec9c4d4b8e269b2a41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.871497 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24972c0535bf41ec9c4d4b8e269b2a41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.873032 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_54eea988fc86453db7e4a99597a5e552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.873448 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54eea988fc86453db7e4a99597a5e552 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.873725 1168 tablet_bootstrap.cc:492] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:17.873843 1168 ts_tablet_manager.cc:1397] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.133s user 0.009s sys 0.002s
I20250115 14:52:17.874403 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f6cc9ac2a299480b990c0a3ddcf355fe" } new_table_name: "default.Table_263c3650ab044fd8b6299e87a94bf394" modify_external_catalogs: true
I20250115 14:52:17.874468 1168 raft_consensus.cc:357] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.874624 1168 raft_consensus.cc:383] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.874675 1168 raft_consensus.cc:738] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.874845 1168 consensus_queue.cc:260] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.875319 1168 ts_tablet_manager.cc:1428] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250115 14:52:17.875583 1168 tablet_bootstrap.cc:492] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:17.880788 1168 tablet_bootstrap.cc:654] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.883251 1167 ts_tablet_manager.cc:1939] T 6e2d55a4e32448fbb98f704dbf1657b9 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:17.883767 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 6e2d55a4e32448fbb98f704dbf1657b9 (table default.Table_f3c56dc604f2471083ae4303c707c7e3 [id=03cd191267784fac938c73c9b363855a]) successfully deleted
I20250115 14:52:17.892279 748 tablet_service.cc:1467] Processing CreateTablet for tablet 7d13ba513b1d4cbbb5c6b5e4c5bd7de3 (DEFAULT_TABLE table=default.Table_12f038b27f044196a4c4f37f1d246025 [id=accf0b6776f0438f9927d0ce35752a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.892707 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d13ba513b1d4cbbb5c6b5e4c5bd7de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.893878 880 tablet_service.cc:1467] Processing CreateTablet for tablet 7d13ba513b1d4cbbb5c6b5e4c5bd7de3 (DEFAULT_TABLE table=default.Table_12f038b27f044196a4c4f37f1d246025 [id=accf0b6776f0438f9927d0ce35752a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.894215 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d13ba513b1d4cbbb5c6b5e4c5bd7de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.895532 889 tablet_service.cc:1467] Processing CreateTablet for tablet 0375eea766464ac195ee97d447687dd6 (DEFAULT_TABLE table=default.Table_cd7cf126bef040e7aac0c1b681b4370d [id=db2289bc93ff4dac9e3e26ecf95630bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.895810 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0375eea766464ac195ee97d447687dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.896625 751 tablet_service.cc:1467] Processing CreateTablet for tablet 0375eea766464ac195ee97d447687dd6 (DEFAULT_TABLE table=default.Table_cd7cf126bef040e7aac0c1b681b4370d [id=db2289bc93ff4dac9e3e26ecf95630bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.896957 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0375eea766464ac195ee97d447687dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.902702 743 tablet_service.cc:1467] Processing CreateTablet for tablet 84c3d37df68245ad8514d114152a70f3 (DEFAULT_TABLE table=default.Table_5c58ab16f3b6495681b201cf7a5652c0 [id=d7357170776e424f92714831b35d61a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.903043 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84c3d37df68245ad8514d114152a70f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.910315 889 tablet_service.cc:1467] Processing CreateTablet for tablet 84c3d37df68245ad8514d114152a70f3 (DEFAULT_TABLE table=default.Table_5c58ab16f3b6495681b201cf7a5652c0 [id=d7357170776e424f92714831b35d61a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.910764 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84c3d37df68245ad8514d114152a70f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.894665 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 7d13ba513b1d4cbbb5c6b5e4c5bd7de3 (DEFAULT_TABLE table=default.Table_12f038b27f044196a4c4f37f1d246025 [id=accf0b6776f0438f9927d0ce35752a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.912031 1164 tablet_bootstrap.cc:492] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.912143 1164 ts_tablet_manager.cc:1397] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.127s user 0.003s sys 0.014s
I20250115 14:52:17.912249 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d13ba513b1d4cbbb5c6b5e4c5bd7de3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.900678 1162 tablet_bootstrap.cc:492] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:17.912611 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 84c3d37df68245ad8514d114152a70f3 (DEFAULT_TABLE table=default.Table_5c58ab16f3b6495681b201cf7a5652c0 [id=d7357170776e424f92714831b35d61a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.912820 1162 ts_tablet_manager.cc:1397] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.127s user 0.016s sys 0.013s
I20250115 14:52:17.912933 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84c3d37df68245ad8514d114152a70f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.913257 743 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea47079a69447e3b9f922526c046a85 (DEFAULT_TABLE table=default.Table_fea785c7867c44bd8271e1705fe4d238 [id=2270b6c1dfd247ba933e34d1f6a7e104]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.913553 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea47079a69447e3b9f922526c046a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.913589 1162 raft_consensus.cc:357] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.913753 1162 raft_consensus.cc:383] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.913815 1162 raft_consensus.cc:738] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.913988 1162 consensus_queue.cc:260] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.914486 1162 ts_tablet_manager.cc:1428] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:17.914736 1162 tablet_bootstrap.cc:492] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:17.916224 1162 tablet_bootstrap.cc:654] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.902462 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 0375eea766464ac195ee97d447687dd6 (DEFAULT_TABLE table=default.Table_cd7cf126bef040e7aac0c1b681b4370d [id=db2289bc93ff4dac9e3e26ecf95630bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.919226 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0375eea766464ac195ee97d447687dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.919515 1164 raft_consensus.cc:357] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.919694 1164 raft_consensus.cc:383] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.919751 1164 raft_consensus.cc:738] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.921301 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea47079a69447e3b9f922526c046a85 (DEFAULT_TABLE table=default.Table_fea785c7867c44bd8271e1705fe4d238 [id=2270b6c1dfd247ba933e34d1f6a7e104]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.921638 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea47079a69447e3b9f922526c046a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.922082 751 tablet_service.cc:1467] Processing CreateTablet for tablet 882a09420b92457e8e26a7c3f81adce3 (DEFAULT_TABLE table=default.Table_765ae278c41246aca70aa158f4ee2a08 [id=4917a6f6e20741189fb8e838ca44e173]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.922219 884 tablet_service.cc:1467] Processing CreateTablet for tablet 882a09420b92457e8e26a7c3f81adce3 (DEFAULT_TABLE table=default.Table_765ae278c41246aca70aa158f4ee2a08 [id=4917a6f6e20741189fb8e838ca44e173]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.922384 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882a09420b92457e8e26a7c3f81adce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.922531 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882a09420b92457e8e26a7c3f81adce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.924887 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ec2a8000f5474f62aae52c8e88e45276
I20250115 14:52:17.925488 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_b2803f567cf849a480c49e93808b1a19" } new_table_name: "default.Table_711c663660094ab5bfd4276b64fbebfa" modify_external_catalogs: true
I20250115 14:52:17.927145 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_19257b7b80b44471bc2848b06e1e25ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:17.927278 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_7b8c967763964a01a95d0d86f202070c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.927646 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19257b7b80b44471bc2848b06e1e25ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.882097 1751 ts_tablet_manager.cc:1918] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:17.928417 1751 log.cc:1198] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/f60e68b411364ada9b89879012d862ea
I20250115 14:52:17.928890 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2a8000f5474f62aae52c8e88e45276 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.929093 1166 tablet_replica.cc:331] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:17.929011 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_beee9856d4a747f48cfdeb296778ab88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:17.929258 1166 raft_consensus.cc:2238] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 1 LEADER]: Raft consensus shutting down.
W20250115 14:52:17.929421 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_beee9856d4a747f48cfdeb296778ab88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.929519 1166 raft_consensus.cc:2267] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.930019 1166 ts_tablet_manager.cc:1905] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.930524 450 master_service.cc:946] ReplaceTablet: received request to replace tablet e49dc97eea474252a5e0ef4af6e015a9 from {username='slave'} at 127.0.0.1:47464
W20250115 14:52:17.931631 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f3c12a0a7eb42c5888961103f7eb178 consensus state has no leader
I20250115 14:52:17.933506 1751 ts_tablet_manager.cc:1939] T f60e68b411364ada9b89879012d862ea P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:17.934834 880 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea47079a69447e3b9f922526c046a85 (DEFAULT_TABLE table=default.Table_fea785c7867c44bd8271e1705fe4d238 [id=2270b6c1dfd247ba933e34d1f6a7e104]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.926616 447 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_a3ebaf4214574dd68c619382cb749409 [id=c66ea162404c4eb0a012128385984399] alter complete (version 1)
I20250115 14:52:17.935914 2915 raft_consensus.cc:491] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:17.935968 452 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 08d9f83287c6450cb30d0d02dc1c6238
I20250115 14:52:17.936039 2915 raft_consensus.cc:513] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:17.927647 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b8c967763964a01a95d0d86f202070c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.936610 2915 leader_election.cc:290] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:17.936635 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 10c9de0f178743f88e1334c5397e4fd3
I20250115 14:52:17.937086 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8589265cc054407b68055bf5665057e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:17.937251 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ae1a9330d9fd4cd5b629af7e1f44ab0e
W20250115 14:52:17.937551 984 leader_election.cc:343] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.938270 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 2560d42c15c64b4c9c0cfe95706b4d9e with delete_type TABLET_DATA_DELETED (replaced by tablet f4536bb3d44b40d2825e1f4e59d8a148) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.938557 882 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.938616 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 08d9f83287c6450cb30d0d02dc1c6238 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.938570 883 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2a8000f5474f62aae52c8e88e45276 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.938889 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 08d9f83287c6450cb30d0d02dc1c6238 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.939076 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c9de0f178743f88e1334c5397e4fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.939108 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c9de0f178743f88e1334c5397e4fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:17.939203 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.939754 1166 ts_tablet_manager.cc:1918] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:17.939894 1166 log.cc:1198] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/ec2a8000f5474f62aae52c8e88e45276
I20250115 14:52:17.940124 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6a2171304c82499ea4dd40cf34994b06" } new_table_name: "default.Table_51e5cdc658b743a1a88eaf7282d7b31f" modify_external_catalogs: true
I20250115 14:52:17.940249 1166 ts_tablet_manager.cc:1939] T ec2a8000f5474f62aae52c8e88e45276 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:17.941357 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a9ce413c298446edb254872574b66f69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.941792 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9ce413c298446edb254872574b66f69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.942389 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_3626af9a8dea40849743366863e8f8f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:17.942826 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3626af9a8dea40849743366863e8f8f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:17.942930 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c3d7415d4479485b9629973554670e05" } new_table_name: "default.Table_9c3096c6cd7249b98b742206b0334ff9" modify_external_catalogs: true
I20250115 14:52:17.943796 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_fc7747535f7b4449867ef0aeb96f88f4" } new_table_name: "default.Table_1bae29e9e7e14396b6d79e0e6fca4e41" modify_external_catalogs: true
I20250115 14:52:17.944212 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c9de0f178743f88e1334c5397e4fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.944329 742 tablet_service.cc:1514] Processing DeleteTablet for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.944164 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 08d9f83287c6450cb30d0d02dc1c6238 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:17.946233 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet f60e68b411364ada9b89879012d862ea (table default.Table_f8719599d09f457681ab1c5bbf617baa [id=ce85b2d5cbb3453e8237d9404cb9f94e]) successfully deleted
I20250115 14:52:17.946288 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2a8000f5474f62aae52c8e88e45276 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:17 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:17.946928 1751 tablet_replica.cc:331] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:17.947165 1751 raft_consensus.cc:2238] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:17.947337 1751 raft_consensus.cc:2267] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:17.947871 1751 ts_tablet_manager.cc:1905] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:17.947794 1164 consensus_queue.cc:260] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.948122 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet ec2a8000f5474f62aae52c8e88e45276 (table default.Table_e79474d461c04307bd06aac8b3a66328 [id=1b0b109197b0474395a710dd740f944f]) successfully deleted
W20250115 14:52:17.948343 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 10c9de0f178743f88e1334c5397e4fd3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 10c9de0f178743f88e1334c5397e4fd3 already in progress: creating tablet
W20250115 14:52:17.948495 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet ae1a9330d9fd4cd5b629af7e1f44ab0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae1a9330d9fd4cd5b629af7e1f44ab0e already in progress: creating tablet
W20250115 14:52:17.948627 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 08d9f83287c6450cb30d0d02dc1c6238 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08d9f83287c6450cb30d0d02dc1c6238 already in progress: creating tablet
I20250115 14:52:17.951674 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8589265cc054407b68055bf5665057e" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:17.952378 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ea47079a69447e3b9f922526c046a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.956518 1164 ts_tablet_manager.cc:1428] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
I20250115 14:52:17.956831 1164 tablet_bootstrap.cc:492] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:17.959205 1164 tablet_bootstrap.cc:654] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.959506 751 tablet_service.cc:1467] Processing CreateTablet for tablet f8ecde3224ee460daf37c78ea9a5b9d7 (DEFAULT_TABLE table=default.Table_19e166ae7bf948e394394e722ff61d97 [id=a5ee80d70ef840d89e82cd6a9470995c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.960350 748 tablet_service.cc:1467] Processing CreateTablet for tablet d249b9f061a34acf8147a961b3b6fb74 (DEFAULT_TABLE table=default.Table_89ac4cd90d424ff9b3d332264f554d0e [id=9435868401074ffb8917d4f8f8a5e998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.961037 742 tablet_service.cc:1467] Processing CreateTablet for tablet 7db2a48abcfd4d33ad9b286270eefe5a (DEFAULT_TABLE table=default.Table_d56c0d67f1514a0d8167c4a83162cbec [id=ce385adc3f09494a840aa89c70150028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.962023 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d249b9f061a34acf8147a961b3b6fb74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.962234 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db2a48abcfd4d33ad9b286270eefe5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.968066 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8ecde3224ee460daf37c78ea9a5b9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.969913 891 tablet_service.cc:1467] Processing CreateTablet for tablet d249b9f061a34acf8147a961b3b6fb74 (DEFAULT_TABLE table=default.Table_89ac4cd90d424ff9b3d332264f554d0e [id=9435868401074ffb8917d4f8f8a5e998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.970243 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d249b9f061a34acf8147a961b3b6fb74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.970366 884 tablet_service.cc:1467] Processing CreateTablet for tablet f8ecde3224ee460daf37c78ea9a5b9d7 (DEFAULT_TABLE table=default.Table_19e166ae7bf948e394394e722ff61d97 [id=a5ee80d70ef840d89e82cd6a9470995c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.970762 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8ecde3224ee460daf37c78ea9a5b9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.972738 1168 tablet_bootstrap.cc:492] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:17.973012 1168 ts_tablet_manager.cc:1397] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.000s
I20250115 14:52:17.973735 1168 raft_consensus.cc:357] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
W20250115 14:52:17.952338 983 leader_election.cc:343] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:17.973919 983 leader_election.cc:304] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:17.974869 2839 raft_consensus.cc:2743] T b8589265cc054407b68055bf5665057e P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:17.965152 1021 tablet_service.cc:1467] Processing CreateTablet for tablet d249b9f061a34acf8147a961b3b6fb74 (DEFAULT_TABLE table=default.Table_89ac4cd90d424ff9b3d332264f554d0e [id=9435868401074ffb8917d4f8f8a5e998]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.975838 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d249b9f061a34acf8147a961b3b6fb74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.976438 1168 raft_consensus.cc:383] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.976500 1168 raft_consensus.cc:738] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.976682 1168 consensus_queue.cc:260] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.977972 1168 ts_tablet_manager.cc:1428] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:17.980788 1168 tablet_bootstrap.cc:492] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:17.982414 1168 tablet_bootstrap.cc:654] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:17.983429 1164 tablet_bootstrap.cc:492] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:17.983554 1164 ts_tablet_manager.cc:1397] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20250115 14:52:17.984213 1164 raft_consensus.cc:357] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.984413 1164 raft_consensus.cc:383] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:17.984467 1164 raft_consensus.cc:738] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:17.986553 1164 consensus_queue.cc:260] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:17.987062 1164 ts_tablet_manager.cc:1428] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:17.987334 1164 tablet_bootstrap.cc:492] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
W20250115 14:52:17.989320 455 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7718729182a340fe9b2dfea5a8321c1f consensus state has no leader
I20250115 14:52:17.969650 1017 tablet_service.cc:1467] Processing CreateTablet for tablet f8ecde3224ee460daf37c78ea9a5b9d7 (DEFAULT_TABLE table=default.Table_19e166ae7bf948e394394e722ff61d97 [id=a5ee80d70ef840d89e82cd6a9470995c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:17.990134 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8ecde3224ee460daf37c78ea9a5b9d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:17.994316 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet b8589265cc054407b68055bf5665057e (table default.Table_2ef4d39c78514fb795a21054fb20be91 [id=ce06e03a310c40f2a3b2883f2bf45b5e]). Expected version 1 got 0
I20250115 14:52:17.997319 454 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 (table default.Table_a3ebaf4214574dd68c619382cb749409 [id=c66ea162404c4eb0a012128385984399]). Expected version 1 got 0
I20250115 14:52:18.000435 1164 tablet_bootstrap.cc:654] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.006152 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet e49dc97eea474252a5e0ef4af6e015a9
I20250115 14:52:18.006548 450 catalog_manager.cc:6596] ReplaceTablet: tablet e49dc97eea474252a5e0ef4af6e015a9 deleted and replaced by tablet 866454761ddb4455b7deb0231d413bc6
I20250115 14:52:17.973737 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 882a09420b92457e8e26a7c3f81adce3 (DEFAULT_TABLE table=default.Table_765ae278c41246aca70aa158f4ee2a08 [id=4917a6f6e20741189fb8e838ca44e173]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.007339 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882a09420b92457e8e26a7c3f81adce3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:18.007403 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b252e48108947cbbb36335499b51c92 consensus state has no leader
W20250115 14:52:18.008070 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e41d8ab30f974bdcba94346460b504a6 consensus state has no leader
I20250115 14:52:18.009527 2915 raft_consensus.cc:491] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.009650 2915 raft_consensus.cc:513] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.010262 2915 leader_election.cc:290] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.011245 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d612b424bea408cb6691c7c200cf3a6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.011731 984 leader_election.cc:343] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.012267 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 1d1df21ab18f45a8adf9fedd22b2b8a9 (table default.Table_a3ebaf4214574dd68c619382cb749409 [id=c66ea162404c4eb0a012128385984399]). Expected version 1 got 0
I20250115 14:52:18.014988 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet e49dc97eea474252a5e0ef4af6e015a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 866454761ddb4455b7deb0231d413bc6) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.015637 751 tablet_service.cc:1514] Processing DeleteTablet for tablet e244689cef6844e2935becd265eecad0 with delete_type TABLET_DATA_DELETED (replaced by tablet 2a1a69b6de5d433f82579b1259548fd2) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.015887 1166 tablet_replica.cc:331] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:18.016050 1166 raft_consensus.cc:2238] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.016191 1166 raft_consensus.cc:2267] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.016520 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a201de8c5f304323ac40334c8221a456"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:18.016873 1166 ts_tablet_manager.cc:1905] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:18.017061 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a201de8c5f304323ac40334c8221a456 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.018282 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_2b33069ab0bb4cfa9d194d732eb8aa8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.018692 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b33069ab0bb4cfa9d194d732eb8aa8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.021778 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6754ab1e6865448fafba9318cd2ea307" } modify_external_catalogs: true
I20250115 14:52:18.021925 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6754ab1e6865448fafba9318cd2ea307" } modify_external_catalogs: true
I20250115 14:52:18.022689 1166 ts_tablet_manager.cc:1918] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.022786 1166 log.cc:1198] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e244689cef6844e2935becd265eecad0
I20250115 14:52:18.023058 1166 ts_tablet_manager.cc:1939] T e244689cef6844e2935becd265eecad0 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:18.024281 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e244689cef6844e2935becd265eecad0 (table default.Table_54e0718fb03d45629ac03649aacf44ba [id=1cdf416758254c679c3c785b101c65e2]) successfully deleted
I20250115 14:52:18.024955 880 tablet_service.cc:1514] Processing DeleteTablet for tablet e6978223c1984a38bec5f2d06a9cf457 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:01 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.028389 751 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9868ffce2b445fa433fd80e6811639 (DEFAULT_TABLE table=default.Table_3f53b19c7d544a56b0b4307eff6da294 [id=1cfd04e44bfe4ca38285e7a5b5bc5bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.028606 883 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9868ffce2b445fa433fd80e6811639 (DEFAULT_TABLE table=default.Table_3f53b19c7d544a56b0b4307eff6da294 [id=1cfd04e44bfe4ca38285e7a5b5bc5bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.028757 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9868ffce2b445fa433fd80e6811639. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.028676 880 tablet_service.cc:1467] Processing CreateTablet for tablet 7db2a48abcfd4d33ad9b286270eefe5a (DEFAULT_TABLE table=default.Table_d56c0d67f1514a0d8167c4a83162cbec [id=ce385adc3f09494a840aa89c70150028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.029240 891 tablet_service.cc:1514] Processing DeleteTablet for tablet e49dc97eea474252a5e0ef4af6e015a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 866454761ddb4455b7deb0231d413bc6) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.029567 1167 tablet_replica.cc:331] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:18.029585 2865 raft_consensus.cc:491] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.029706 2865 raft_consensus.cc:513] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.029816 1167 raft_consensus.cc:2238] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.030037 1167 raft_consensus.cc:2267] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.030239 2865 leader_election.cc:290] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:18.030376 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d612b424bea408cb6691c7c200cf3a6" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:18.030655 1167 ts_tablet_manager.cc:1905] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:18.030851 983 leader_election.cc:343] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.030967 983 leader_election.cc:304] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.031359 2915 raft_consensus.cc:2743] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.032989 775 raft_consensus.cc:3054] T a5e97279185541b992340c24ede01656 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:18.033103 1046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1df21ab18f45a8adf9fedd22b2b8a9" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:18.032119 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 7db2a48abcfd4d33ad9b286270eefe5a (DEFAULT_TABLE table=default.Table_d56c0d67f1514a0d8167c4a83162cbec [id=ce385adc3f09494a840aa89c70150028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.034397 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db2a48abcfd4d33ad9b286270eefe5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.035622 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d1df21ab18f45a8adf9fedd22b2b8a9" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:18.036245 710 leader_election.cc:304] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:18.036983 2865 raft_consensus.cc:2743] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.043053 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 9a9868ffce2b445fa433fd80e6811639 (DEFAULT_TABLE table=default.Table_3f53b19c7d544a56b0b4307eff6da294 [id=1cfd04e44bfe4ca38285e7a5b5bc5bd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.045562 751 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c (DEFAULT_TABLE table=default.Table_6754ab1e6865448fafba9318cd2ea307 [id=f68fd2eb9372408b85a827a5bc1abb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.045904 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:18.046720 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c3f78abe3a6049cb90d45e5f9e61b061 consensus state has no leader
I20250115 14:52:18.050486 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c (DEFAULT_TABLE table=default.Table_6754ab1e6865448fafba9318cd2ea307 [id=f68fd2eb9372408b85a827a5bc1abb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.051482 891 tablet_service.cc:1467] Processing CreateTablet for tablet acc43b7b402249339f6bf213e35e35b9 (DEFAULT_TABLE table=default.Table_2c0b01c3b5394ebbac4a5e930276ed6c [id=2a505c2b70024b21b8b42727cab3d028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.051981 1028 tablet_service.cc:1467] Processing CreateTablet for tablet e641ebc980554a2388e9f9312a8921b2 (DEFAULT_TABLE table=default.Table_2cb376913356450792ee862605884f0d [id=055fa001545d4dc2a6d2b1c5522120f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.053049 877 tablet_service.cc:1467] Processing CreateTablet for tablet 24b0327b26fb4468816b91fe33c832f3 (DEFAULT_TABLE table=default.Table_1a6fffc23de14381b074fafe7e270032 [id=600319966c65450abbe9cb10563db102]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.053251 1012 tablet_service.cc:1467] Processing CreateTablet for tablet acc43b7b402249339f6bf213e35e35b9 (DEFAULT_TABLE table=default.Table_2c0b01c3b5394ebbac4a5e930276ed6c [id=2a505c2b70024b21b8b42727cab3d028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.053483 1010 tablet_service.cc:1467] Processing CreateTablet for tablet 24b0327b26fb4468816b91fe33c832f3 (DEFAULT_TABLE table=default.Table_1a6fffc23de14381b074fafe7e270032 [id=600319966c65450abbe9cb10563db102]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.053668 748 tablet_service.cc:1467] Processing CreateTablet for tablet acc43b7b402249339f6bf213e35e35b9 (DEFAULT_TABLE table=default.Table_2c0b01c3b5394ebbac4a5e930276ed6c [id=2a505c2b70024b21b8b42727cab3d028]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.053900 1011 tablet_service.cc:1467] Processing CreateTablet for tablet dd2e8aa0012444a489087213f9111f19 (DEFAULT_TABLE table=default.Table_ab770caafef447218be7ae6edcd11e57 [id=d9dac5a3219b48419840e02c2a26c1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.053999 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc43b7b402249339f6bf213e35e35b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.054848 882 tablet_service.cc:1467] Processing CreateTablet for tablet dd2e8aa0012444a489087213f9111f19 (DEFAULT_TABLE table=default.Table_ab770caafef447218be7ae6edcd11e57 [id=d9dac5a3219b48419840e02c2a26c1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.055284 742 tablet_service.cc:1467] Processing CreateTablet for tablet 24b0327b26fb4468816b91fe33c832f3 (DEFAULT_TABLE table=default.Table_1a6fffc23de14381b074fafe7e270032 [id=600319966c65450abbe9cb10563db102]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.056224 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24b0327b26fb4468816b91fe33c832f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.056505 737 tablet_service.cc:1467] Processing CreateTablet for tablet dd2e8aa0012444a489087213f9111f19 (DEFAULT_TABLE table=default.Table_ab770caafef447218be7ae6edcd11e57 [id=d9dac5a3219b48419840e02c2a26c1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.056818 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2e8aa0012444a489087213f9111f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.058053 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.058211 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24b0327b26fb4468816b91fe33c832f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.060824 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e641ebc980554a2388e9f9312a8921b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.060989 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2e8aa0012444a489087213f9111f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.061488 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc43b7b402249339f6bf213e35e35b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.065405 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24b0327b26fb4468816b91fe33c832f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.065608 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd2e8aa0012444a489087213f9111f19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.066982 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acc43b7b402249339f6bf213e35e35b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.068480 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9868ffce2b445fa433fd80e6811639. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.072682 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a9868ffce2b445fa433fd80e6811639. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.073549 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c
I20250115 14:52:18.074134 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.074633 1027 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:18.075058 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c already in progress: creating tablet
I20250115 14:52:18.077337 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7db2a48abcfd4d33ad9b286270eefe5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.089413 2915 raft_consensus.cc:491] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.089543 2915 raft_consensus.cc:513] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.090108 2915 leader_election.cc:290] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.090747 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fddbd9ba14443a91865387e1d1c238" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:18.091324 983 leader_election.cc:343] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.091878 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fddbd9ba14443a91865387e1d1c238" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.092384 984 leader_election.cc:343] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.092474 984 leader_election.cc:304] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.092846 2915 raft_consensus.cc:2743] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.093515 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.097990 889 tablet_service.cc:1467] Processing CreateTablet for tablet e641ebc980554a2388e9f9312a8921b2 (DEFAULT_TABLE table=default.Table_2cb376913356450792ee862605884f0d [id=055fa001545d4dc2a6d2b1c5522120f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.098438 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e641ebc980554a2388e9f9312a8921b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.104312 884 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c (DEFAULT_TABLE table=default.Table_6754ab1e6865448fafba9318cd2ea307 [id=f68fd2eb9372408b85a827a5bc1abb73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.104684 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f8b03e76d9c4cbf9cd3b4bed6119b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.108430 743 tablet_service.cc:1514] Processing DeleteTablet for tablet e49dc97eea474252a5e0ef4af6e015a9 with delete_type TABLET_DATA_DELETED (replaced by tablet 866454761ddb4455b7deb0231d413bc6) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.108764 1166 tablet_replica.cc:331] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:18.108956 1166 raft_consensus.cc:2238] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.109102 1166 raft_consensus.cc:2267] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.109844 1166 ts_tablet_manager.cc:1905] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.112020 1162 tablet_bootstrap.cc:492] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.112159 1162 ts_tablet_manager.cc:1397] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.197s user 0.027s sys 0.028s
I20250115 14:52:18.112907 1162 raft_consensus.cc:357] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.113106 1162 raft_consensus.cc:383] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.113166 1162 raft_consensus.cc:738] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.113345 1162 consensus_queue.cc:260] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.113816 1162 ts_tablet_manager.cc:1428] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:18.112987 742 tablet_service.cc:1467] Processing CreateTablet for tablet 6846983c1b5c456f90292d5f5b600426 (DEFAULT_TABLE table=default.Table_380c0dd6df914022a741ea83ed384911 [id=190589a22d424365a7f858a214fd0f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.115556 1166 ts_tablet_manager.cc:1918] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.115669 1166 log.cc:1198] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/e49dc97eea474252a5e0ef4af6e015a9
I20250115 14:52:18.115955 1166 ts_tablet_manager.cc:1939] T e49dc97eea474252a5e0ef4af6e015a9 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:18.116406 1162 tablet_bootstrap.cc:492] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.116601 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 6846983c1b5c456f90292d5f5b600426 (DEFAULT_TABLE table=default.Table_380c0dd6df914022a741ea83ed384911 [id=190589a22d424365a7f858a214fd0f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.116923 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6846983c1b5c456f90292d5f5b600426. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.117918 1162 tablet_bootstrap.cc:654] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.118148 1751 ts_tablet_manager.cc:1918] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.118250 1751 log.cc:1198] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/e180e37a1d5d46d78380323662b80f85
I20250115 14:52:18.118543 1751 ts_tablet_manager.cc:1939] T e180e37a1d5d46d78380323662b80f85 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:18.119228 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1371) took 1119 ms. Trace:
I20250115 14:52:18.119302 1751 rpcz_store.cc:276] 0115 14:52:16.999995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.000049 (+ 54us) service_pool.cc:225] Handling call
0115 14:52:17.946694 (+946645us) ts_tablet_manager.cc:1145] Deleting tablet e180e37a1d5d46d78380323662b80f85
0115 14:52:17.946891 (+ 197us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.052576 (+105685us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.118056 (+ 65480us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.119211 (+ 1155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":946341}
I20250115 14:52:18.115571 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6846983c1b5c456f90292d5f5b600426. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.120514 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet e49dc97eea474252a5e0ef4af6e015a9 (table default.Table_4f00ef2ef63d4f77929a245aa9bff035 [id=8502b5fe264f4fb5a89bd4a91949c90b]) successfully deleted
I20250115 14:52:18.121978 751 tablet_service.cc:1467] Processing CreateTablet for tablet e641ebc980554a2388e9f9312a8921b2 (DEFAULT_TABLE table=default.Table_2cb376913356450792ee862605884f0d [id=055fa001545d4dc2a6d2b1c5522120f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.122311 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e641ebc980554a2388e9f9312a8921b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.124583 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet c14836b97c3247b39ff9072c12e642e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:06 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.124926 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet e180e37a1d5d46d78380323662b80f85 (table default.Table_ae2b381e4eb84e6788f5d29ffc7a72f7 [id=a2079a8b5634463b83a77df10fd713a7]) successfully deleted
I20250115 14:52:18.129117 1168 tablet_bootstrap.cc:492] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:18.129316 1168 ts_tablet_manager.cc:1397] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.002s
I20250115 14:52:18.129992 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9f03b68742a7468d8e2f0e5e4af9f094" } modify_external_catalogs: true
I20250115 14:52:18.130175 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_9f03b68742a7468d8e2f0e5e4af9f094" } modify_external_catalogs: true
I20250115 14:52:18.131520 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_cca21d06682c49a3a8f1aa144af69dc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.132055 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cca21d06682c49a3a8f1aa144af69dc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.134545 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_5dc7a5be3fe34c81892c07ff7348cdf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.134981 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5dc7a5be3fe34c81892c07ff7348cdf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.135975 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_315961ed120f42f18e32f4612f7d30ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.136408 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_315961ed120f42f18e32f4612f7d30ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.137712 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5beff98980534f9694b89e4fa022ab17" } new_table_name: "default.Table_bb91482d040149aa81efe1bee175500d" modify_external_catalogs: true
I20250115 14:52:18.140717 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7a882c6c77fd4d198c18380628c8ef1a" } modify_external_catalogs: true
I20250115 14:52:18.140875 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_7a882c6c77fd4d198c18380628c8ef1a" } modify_external_catalogs: true
I20250115 14:52:18.140920 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_572fd8a32b744555a1e45e1321f35352" } modify_external_catalogs: true
I20250115 14:52:18.141062 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_572fd8a32b744555a1e45e1321f35352" } modify_external_catalogs: true
I20250115 14:52:18.140765 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f4904239f07b44888c5b08324a52aa5b" } modify_external_catalogs: true
I20250115 14:52:18.141891 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_f4904239f07b44888c5b08324a52aa5b" } modify_external_catalogs: true
I20250115 14:52:18.145941 1164 tablet_bootstrap.cc:492] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:18.146107 1164 ts_tablet_manager.cc:1397] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.159s user 0.002s sys 0.006s
I20250115 14:52:18.146898 1164 raft_consensus.cc:357] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.147104 1164 raft_consensus.cc:383] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.147186 1164 raft_consensus.cc:738] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.147459 1164 consensus_queue.cc:260] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.148487 1164 ts_tablet_manager.cc:1428] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250115 14:52:18.148782 1164 tablet_bootstrap.cc:492] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:18.151113 1164 tablet_bootstrap.cc:654] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.153007 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet a98bf19b5df841409887a2666c83b9b4
I20250115 14:52:18.154095 742 tablet_service.cc:1514] Processing DeleteTablet for tablet a98bf19b5df841409887a2666c83b9b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:18.154858 441 catalog_manager.cc:4839] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet a98bf19b5df841409887a2666c83b9b4 because the tablet was not found. No further retry: Not found: Tablet not found: a98bf19b5df841409887a2666c83b9b4
I20250115 14:52:18.154850 1011 tablet_service.cc:1514] Processing DeleteTablet for tablet a98bf19b5df841409887a2666c83b9b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.158217 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 9494c0c57be8499a883888d067c7db77 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:18.160178 883 tablet_service.cc:1514] Processing DeleteTablet for tablet a98bf19b5df841409887a2666c83b9b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.160815 1751 tablet_replica.cc:331] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:18.161007 1751 raft_consensus.cc:2238] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.161163 1751 raft_consensus.cc:2267] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.161558 1168 raft_consensus.cc:357] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.161681 1751 ts_tablet_manager.cc:1905] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.161828 1168 raft_consensus.cc:383] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.161926 1168 raft_consensus.cc:738] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.162182 1168 consensus_queue.cc:260] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.162717 1168 ts_tablet_manager.cc:1428] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20250115 14:52:18.170383 446 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 246720dbb4ed4216ac87fb4a3df79e2c (table default.Table_daaa2755fe9d4f54ae6ff33ae452ec05 [id=8c0229e6f6884ea7be3417678bb27377]). Expected version 1 got 0
W20250115 14:52:18.180902 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 368de0e78e334d82a588a9ed89dc4b27 consensus state has no leader
I20250115 14:52:18.182550 742 tablet_service.cc:1467] Processing CreateTablet for tablet 177103c75cdc48bf9c3bfefa8f0bd6a9 (DEFAULT_TABLE table=default.Table_69731e3d99894dfa97e738208c012651 [id=b4d219a1f9204b29bdc89821f25e8e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.184168 743 tablet_service.cc:1467] Processing CreateTablet for tablet 80bb289c47f44894a0be1cc20a43c0f3 (DEFAULT_TABLE table=default.Table_9c1c6dca23d34c0090f29d79b5d879fe [id=9466a57d8f3f4db4bb601422bdedc5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.184562 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80bb289c47f44894a0be1cc20a43c0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.185213 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 80bb289c47f44894a0be1cc20a43c0f3 (DEFAULT_TABLE table=default.Table_9c1c6dca23d34c0090f29d79b5d879fe [id=9466a57d8f3f4db4bb601422bdedc5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.185559 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80bb289c47f44894a0be1cc20a43c0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.186513 2839 raft_consensus.cc:491] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.186641 2839 raft_consensus.cc:513] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.186751 748 tablet_service.cc:1467] Processing CreateTablet for tablet 7823219fe99e40d4b40087b066a3c239 (DEFAULT_TABLE table=default.Table_8aae5406185442c5aadb27f03012605f [id=5b80617e1ebf4e79a162506fb46774b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.187075 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7823219fe99e40d4b40087b066a3c239. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.187218 2839 leader_election.cc:290] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.188509 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6203efb862ea4962a6d09f3e4306a92f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.189160 984 leader_election.cc:343] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.192180 877 tablet_service.cc:1467] Processing CreateTablet for tablet 80bb289c47f44894a0be1cc20a43c0f3 (DEFAULT_TABLE table=default.Table_9c1c6dca23d34c0090f29d79b5d879fe [id=9466a57d8f3f4db4bb601422bdedc5af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.192229 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6203efb862ea4962a6d09f3e4306a92f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:18.192873 983 leader_election.cc:343] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.192987 983 leader_election.cc:304] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.193452 2839 raft_consensus.cc:2743] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.194114 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80bb289c47f44894a0be1cc20a43c0f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.194350 448 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fc7d1caa01d349dcb4ffef0c5d7135f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.194923 448 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc7d1caa01d349dcb4ffef0c5d7135f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.196451 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_19880e28f59446b0a4b4a8cc082428b7" } new_table_name: "default.Table_fa08ff841f384c57bcfb0511c559f1df" modify_external_catalogs: true
I20250115 14:52:18.197777 1168 tablet_bootstrap.cc:492] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:18.202332 883 tablet_service.cc:1467] Processing CreateTablet for tablet 177103c75cdc48bf9c3bfefa8f0bd6a9 (DEFAULT_TABLE table=default.Table_69731e3d99894dfa97e738208c012651 [id=b4d219a1f9204b29bdc89821f25e8e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.202675 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 177103c75cdc48bf9c3bfefa8f0bd6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.204670 2549 raft_consensus.cc:491] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.204779 2549 raft_consensus.cc:513] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.205296 2549 leader_election.cc:290] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
W20250115 14:52:18.206296 847 leader_election.cc:343] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: e6978223c1984a38bec5f2d06a9cf457
I20250115 14:52:18.208910 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6978223c1984a38bec5f2d06a9cf457" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.209992 848 leader_election.cc:343] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.210106 848 leader_election.cc:304] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.210306 2549 raft_consensus.cc:2743] T e6978223c1984a38bec5f2d06a9cf457 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.210613 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_62ac146a58da4b3b8ec38b3301c60339"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.211179 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62ac146a58da4b3b8ec38b3301c60339 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.211277 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 177103c75cdc48bf9c3bfefa8f0bd6a9 (DEFAULT_TABLE table=default.Table_69731e3d99894dfa97e738208c012651 [id=b4d219a1f9204b29bdc89821f25e8e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.211685 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 177103c75cdc48bf9c3bfefa8f0bd6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.212636 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a79f830cc62f4025bffbccefea8be344"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.213145 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a79f830cc62f4025bffbccefea8be344 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.213771 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 177103c75cdc48bf9c3bfefa8f0bd6a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.214768 454 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_00fd62cc66984eecb416652888d31733" } new_table_name: "default.Table_28ffe6444e7e442fb2109a5104a98b41" modify_external_catalogs: true
I20250115 14:52:18.216076 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_314d89eab6294557aa2f54ffca8925f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.216665 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_314d89eab6294557aa2f54ffca8925f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.222599 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_6a6ec52dc25c446a81de6c87b5b0a605"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.223146 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a6ec52dc25c446a81de6c87b5b0a605 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.224953 1168 tablet_bootstrap.cc:654] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.225551 877 tablet_service.cc:1514] Processing DeleteTablet for tablet f3280e8ba6db4446aa3e3bfe2836abfa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:03 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.230816 2991 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:40227 every 8 connection(s)
I20250115 14:52:18.233155 2631 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:40227
I20250115 14:52:18.235960 2631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/master-2/data/info.pb
I20250115 14:52:18.236374 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 416bcee62aa34c4b91cc687b22979b35 (DEFAULT_TABLE table=default.Table_ca1016e060984b6eb5d54a07aad3e076 [id=809ce0dbd50d413ab296a8b8eb922a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.236995 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416bcee62aa34c4b91cc687b22979b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.236938 877 tablet_service.cc:1467] Processing CreateTablet for tablet 416bcee62aa34c4b91cc687b22979b35 (DEFAULT_TABLE table=default.Table_ca1016e060984b6eb5d54a07aad3e076 [id=809ce0dbd50d413ab296a8b8eb922a34]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:18.239573 439 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet c487e956367845299e20cea05ce52987 (table default.Table_61c9adac2720423b8be3226a806f6547 [id=9e76d99bc062464580862b5b4bc023ce], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet c487e956367845299e20cea05ce52987 consensus state has no leader
I20250115 14:52:18.239876 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416bcee62aa34c4b91cc687b22979b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.240041 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-task__Z8ae/build/asan/bin/kudu as pid 2631
I20250115 14:52:18.245702 2774 tablet.cc:1721] T 4a79f880415a4b16aabdbb7cd989efdb P c238c037a1c74725a001e2b4a1ff4304: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:18.246176 2849 tablet.cc:1721] T 4a79f880415a4b16aabdbb7cd989efdb P 73c86158cc2641f182ea50a06634592d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:18.258453 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 70167f772ef246f2b1b5fdb380d6f4da
I20250115 14:52:18.267541 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 70167f772ef246f2b1b5fdb380d6f4da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.268209 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 70167f772ef246f2b1b5fdb380d6f4da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.270004 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 70167f772ef246f2b1b5fdb380d6f4da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.270272 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 69c0368e67624a6e8d1f6a4a6275466b
W20250115 14:52:18.270555 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 70167f772ef246f2b1b5fdb380d6f4da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70167f772ef246f2b1b5fdb380d6f4da already in progress: creating tablet
I20250115 14:52:18.270980 883 tablet_service.cc:1467] Processing CreateTablet for tablet 7823219fe99e40d4b40087b066a3c239 (DEFAULT_TABLE table=default.Table_8aae5406185442c5aadb27f03012605f [id=5b80617e1ebf4e79a162506fb46774b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.271178 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 69c0368e67624a6e8d1f6a4a6275466b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.271335 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7823219fe99e40d4b40087b066a3c239. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.271629 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 69c0368e67624a6e8d1f6a4a6275466b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.271742 1010 tablet_service.cc:1514] Processing DeleteTablet for tablet 69c0368e67624a6e8d1f6a4a6275466b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
W20250115 14:52:18.273936 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 69c0368e67624a6e8d1f6a4a6275466b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69c0368e67624a6e8d1f6a4a6275466b already in progress: creating tablet
I20250115 14:52:18.280004 1167 ts_tablet_manager.cc:1918] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.280118 1167 log.cc:1198] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/a300a76b8f7942dea648a20cbb3d5368
I20250115 14:52:18.280403 1167 ts_tablet_manager.cc:1939] T a300a76b8f7942dea648a20cbb3d5368 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:18.281992 1168 tablet_bootstrap.cc:492] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:18.282187 1168 ts_tablet_manager.cc:1397] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.084s user 0.000s sys 0.003s
I20250115 14:52:18.282873 742 tablet_service.cc:1467] Processing CreateTablet for tablet d6b34363b96e431683fff2b051dbc232 (DEFAULT_TABLE table=default.Table_e9987ad74ce74ebbb8467da42a76c854 [id=492dddba05444d28bc26c538fdf9b008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.282878 1168 raft_consensus.cc:357] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.283041 1168 raft_consensus.cc:383] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.283087 1168 raft_consensus.cc:738] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.283281 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b34363b96e431683fff2b051dbc232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.283294 1168 consensus_queue.cc:260] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.283720 1168 ts_tablet_manager.cc:1428] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250115 14:52:18.288805 880 tablet_service.cc:1467] Processing CreateTablet for tablet d6b34363b96e431683fff2b051dbc232 (DEFAULT_TABLE table=default.Table_e9987ad74ce74ebbb8467da42a76c854 [id=492dddba05444d28bc26c538fdf9b008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.288937 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 7823219fe99e40d4b40087b066a3c239 (DEFAULT_TABLE table=default.Table_8aae5406185442c5aadb27f03012605f [id=5b80617e1ebf4e79a162506fb46774b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.289140 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b34363b96e431683fff2b051dbc232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.289300 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7823219fe99e40d4b40087b066a3c239. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.293052 1011 tablet_service.cc:1467] Processing CreateTablet for tablet d6b34363b96e431683fff2b051dbc232 (DEFAULT_TABLE table=default.Table_e9987ad74ce74ebbb8467da42a76c854 [id=492dddba05444d28bc26c538fdf9b008]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.293831 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b34363b96e431683fff2b051dbc232. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.295149 742 tablet_service.cc:1467] Processing CreateTablet for tablet fc465e2526444b378af8ee978fff4d54 (DEFAULT_TABLE table=default.Table_e29cb0a0a76046cab39de4bf04512200 [id=64678b18e5bd42c28d03b195f3603d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.295568 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc465e2526444b378af8ee978fff4d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.302659 882 tablet_service.cc:1467] Processing CreateTablet for tablet fc465e2526444b378af8ee978fff4d54 (DEFAULT_TABLE table=default.Table_e29cb0a0a76046cab39de4bf04512200 [id=64678b18e5bd42c28d03b195f3603d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.302994 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc465e2526444b378af8ee978fff4d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.307358 889 tablet_service.cc:1467] Processing CreateTablet for tablet 6846983c1b5c456f90292d5f5b600426 (DEFAULT_TABLE table=default.Table_380c0dd6df914022a741ea83ed384911 [id=190589a22d424365a7f858a214fd0f5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.307673 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6846983c1b5c456f90292d5f5b600426. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.309816 1010 tablet_service.cc:1467] Processing CreateTablet for tablet fc465e2526444b378af8ee978fff4d54 (DEFAULT_TABLE table=default.Table_e29cb0a0a76046cab39de4bf04512200 [id=64678b18e5bd42c28d03b195f3603d7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.310169 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc465e2526444b378af8ee978fff4d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.310948 743 tablet_service.cc:1467] Processing CreateTablet for tablet a98bf19b5df841409887a2666c83b9b4 (DEFAULT_TABLE table=default.Table_9f03b68742a7468d8e2f0e5e4af9f094 [id=b60751516a8a405f929fbd635bf3c1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.311275 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98bf19b5df841409887a2666c83b9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.312235 2998 tablet.cc:1721] T 4a79f880415a4b16aabdbb7cd989efdb P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250115 14:52:18.318257 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 16a88475f9ff42c1952e5a14ec852e90
I20250115 14:52:18.318861 447 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_3626af9a8dea40849743366863e8f8f9" } new_table_name: "default.Table_ebb552ba0a2d48fd9270ab5f8bf89f06" modify_external_catalogs: true
I20250115 14:52:18.320374 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_263c3650ab044fd8b6299e87a94bf394" } new_table_name: "default.Table_31f3f59e32b7439a9d3808852c845079" modify_external_catalogs: true
I20250115 14:52:18.321133 1162 tablet_bootstrap.cc:492] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.321316 1162 ts_tablet_manager.cc:1397] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.205s user 0.061s sys 0.039s
I20250115 14:52:18.322276 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a88475f9ff42c1952e5a14ec852e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.322345 1028 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a88475f9ff42c1952e5a14ec852e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.322124 1162 raft_consensus.cc:357] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.322675 1162 raft_consensus.cc:383] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.322788 1162 raft_consensus.cc:738] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.323076 742 tablet_service.cc:1514] Processing DeleteTablet for tablet 16a88475f9ff42c1952e5a14ec852e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.323042 1162 consensus_queue.cc:260] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.323536 1162 ts_tablet_manager.cc:1428] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:18.323820 1162 tablet_bootstrap.cc:492] T 4655600323f944969cfc4eaf5bbbb684 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.324249 891 tablet_service.cc:1467] Processing CreateTablet for tablet a98bf19b5df841409887a2666c83b9b4 (DEFAULT_TABLE table=default.Table_9f03b68742a7468d8e2f0e5e4af9f094 [id=b60751516a8a405f929fbd635bf3c1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.324626 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98bf19b5df841409887a2666c83b9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.325737 1028 tablet_service.cc:1467] Processing CreateTablet for tablet a98bf19b5df841409887a2666c83b9b4 (DEFAULT_TABLE table=default.Table_9f03b68742a7468d8e2f0e5e4af9f094 [id=b60751516a8a405f929fbd635bf3c1fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.327322 1164 tablet_bootstrap.cc:492] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:18.327457 1164 ts_tablet_manager.cc:1397] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.179s user 0.001s sys 0.001s
I20250115 14:52:18.328157 1164 raft_consensus.cc:357] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.328352 1164 raft_consensus.cc:383] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.328406 1164 raft_consensus.cc:738] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
W20250115 14:52:18.328696 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 16a88475f9ff42c1952e5a14ec852e90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16a88475f9ff42c1952e5a14ec852e90 already in progress: creating tablet
I20250115 14:52:18.330088 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a98bf19b5df841409887a2666c83b9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.330840 1162 tablet_bootstrap.cc:654] T 4655600323f944969cfc4eaf5bbbb684 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.331069 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 14e9dbd9a12e49a9af94f12c50d9b3a9 (DEFAULT_TABLE table=default.Table_27c8babf31ef4f10b816e123b4efd9d1 [id=4f6f39b0232b4662bf182b17b38697ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.332949 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e9dbd9a12e49a9af94f12c50d9b3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.338171 742 tablet_service.cc:1467] Processing CreateTablet for tablet 14e9dbd9a12e49a9af94f12c50d9b3a9 (DEFAULT_TABLE table=default.Table_27c8babf31ef4f10b816e123b4efd9d1 [id=4f6f39b0232b4662bf182b17b38697ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.338568 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e9dbd9a12e49a9af94f12c50d9b3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.341463 751 tablet_service.cc:1467] Processing CreateTablet for tablet 416bcee62aa34c4b91cc687b22979b35 (DEFAULT_TABLE table=default.Table_ca1016e060984b6eb5d54a07aad3e076 [id=809ce0dbd50d413ab296a8b8eb922a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.341697 748 tablet_service.cc:1467] Processing CreateTablet for tablet b210ae67631742fab37572ed122c8280 (DEFAULT_TABLE table=default.Table_ee552d095ba643fd8ffde9fc50ca565c [id=5bfdc9af0cac413494c581976fefc2aa]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:18.344369 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b5a476872b254f6989640209077bf38c consensus state has no leader
I20250115 14:52:18.345307 1164 consensus_queue.cc:260] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.345788 449 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet a5e97279185541b992340c24ede01656 (table default.Table_a8391687956e4259a71808f267cb4ff0 [id=361ed20953ab4e79b1112ae76b95a423]). Expected version 3 got 0
I20250115 14:52:18.346449 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_15a9cae29c1f4a8b929fc473c18e3cea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.346906 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15a9cae29c1f4a8b929fc473c18e3cea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.341765 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416bcee62aa34c4b91cc687b22979b35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.349058 1162 tablet_bootstrap.cc:492] T 4655600323f944969cfc4eaf5bbbb684 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.349118 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b210ae67631742fab37572ed122c8280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.349184 1162 ts_tablet_manager.cc:1397] T 4655600323f944969cfc4eaf5bbbb684 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20250115 14:52:18.348183 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b55c04335d174c5982d87569b8c53422"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.349485 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b55c04335d174c5982d87569b8c53422 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.349815 1162 raft_consensus.cc:357] T 4655600323f944969cfc4eaf5bbbb684 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.349989 1162 raft_consensus.cc:383] T 4655600323f944969cfc4eaf5bbbb684 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.350037 1162 raft_consensus.cc:738] T 4655600323f944969cfc4eaf5bbbb684 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.350242 1162 consensus_queue.cc:260] T 4655600323f944969cfc4eaf5bbbb684 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.350694 1162 ts_tablet_manager.cc:1428] T 4655600323f944969cfc4eaf5bbbb684 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250115 14:52:18.350957 1162 tablet_bootstrap.cc:492] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.352165 1162 tablet_bootstrap.cc:654] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.352962 2748 raft_consensus.cc:491] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.353094 2748 raft_consensus.cc:513] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.353580 2748 leader_election.cc:290] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:18.353739 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1368) took 1338 ms. Trace:
I20250115 14:52:18.353821 1167 rpcz_store.cc:276] 0115 14:52:17.014891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.014971 (+ 80us) service_pool.cc:225] Handling call
0115 14:52:18.029511 (+1014540us) ts_tablet_manager.cc:1145] Deleting tablet a300a76b8f7942dea648a20cbb3d5368
0115 14:52:18.029520 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.096738 (+ 67218us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.279914 (+183176us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.353720 (+ 73806us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70271,"spinlock_wait_cycles":1792,"tablet-delete.queue_time_us":1014422}
I20250115 14:52:18.354431 1167 tablet_replica.cc:331] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:18.354635 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet a300a76b8f7942dea648a20cbb3d5368 (table default.Table_0034e4c667b44bbb83a5a93e321e5113 [id=f6bae7686ebc45488f5eb27a554d28a3]) successfully deleted
I20250115 14:52:18.355165 2748 raft_consensus.cc:491] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.355233 1167 raft_consensus.cc:2238] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.355265 2748 raft_consensus.cc:513] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.355758 2748 leader_election.cc:290] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:18.356417 877 tablet_service.cc:1467] Processing CreateTablet for tablet 14e9dbd9a12e49a9af94f12c50d9b3a9 (DEFAULT_TABLE table=default.Table_27c8babf31ef4f10b816e123b4efd9d1 [id=4f6f39b0232b4662bf182b17b38697ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.356729 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e9dbd9a12e49a9af94f12c50d9b3a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.356659 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18f23815196047628d562da7312b010f" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:18.357378 1167 raft_consensus.cc:2267] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:18.357398 848 leader_election.cc:343] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.357928 1167 ts_tablet_manager.cc:1905] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.359454 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a48a0af9422f43cf86531665c0fa1aee" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.359840 848 leader_election.cc:343] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.360056 1168 tablet_bootstrap.cc:492] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
W20250115 14:52:18.360414 847 leader_election.cc:343] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: a48a0af9422f43cf86531665c0fa1aee
I20250115 14:52:18.360508 847 leader_election.cc:304] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.361068 2549 raft_consensus.cc:2743] T a48a0af9422f43cf86531665c0fa1aee P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.361318 1168 tablet_bootstrap.cc:654] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.361858 1751 ts_tablet_manager.cc:1918] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.361958 1751 log.cc:1198] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/c6cb8dfa7c42480bbd139d5acf059a57
I20250115 14:52:18.362218 1751 ts_tablet_manager.cc:1939] T c6cb8dfa7c42480bbd139d5acf059a57 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:18.366462 1167 ts_tablet_manager.cc:1918] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.366560 1167 log.cc:1198] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/0fd94e4a7fef44f5be462e7a77e8a123
I20250115 14:52:18.366815 1167 ts_tablet_manager.cc:1939] T 0fd94e4a7fef44f5be462e7a77e8a123 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:18.367079 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1366) took 1352 ms. Trace:
I20250115 14:52:18.367156 1167 rpcz_store.cc:276] 0115 14:52:17.014689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.015119 (+ 430us) service_pool.cc:225] Handling call
0115 14:52:18.354385 (+1339266us) ts_tablet_manager.cc:1145] Deleting tablet 0fd94e4a7fef44f5be462e7a77e8a123
0115 14:52:18.354395 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.363332 (+ 8937us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.366385 (+ 3053us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.367066 (+ 681us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1339141}
I20250115 14:52:18.367483 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 0fd94e4a7fef44f5be462e7a77e8a123 (table default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]) successfully deleted
I20250115 14:52:18.377214 889 tablet_service.cc:1467] Processing CreateTablet for tablet b210ae67631742fab37572ed122c8280 (DEFAULT_TABLE table=default.Table_ee552d095ba643fd8ffde9fc50ca565c [id=5bfdc9af0cac413494c581976fefc2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.377578 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b210ae67631742fab37572ed122c8280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.377542 1014 tablet_service.cc:1467] Processing CreateTablet for tablet b210ae67631742fab37572ed122c8280 (DEFAULT_TABLE table=default.Table_ee552d095ba643fd8ffde9fc50ca565c [id=5bfdc9af0cac413494c581976fefc2aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.377864 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b210ae67631742fab37572ed122c8280. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.380415 2748 raft_consensus.cc:491] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.380530 2748 raft_consensus.cc:513] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.381103 2748 leader_election.cc:290] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:18.383116 1164 ts_tablet_manager.cc:1428] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20250115 14:52:18.384565 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2560d42c15c64b4c9c0cfe95706b4d9e" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.385056 848 leader_election.cc:343] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
W20250115 14:52:18.391275 847 leader_election.cc:343] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 18f23815196047628d562da7312b010f
I20250115 14:52:18.391814 847 leader_election.cc:304] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.392968 2549 raft_consensus.cc:2743] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:18.395361 847 leader_election.cc:343] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 2560d42c15c64b4c9c0cfe95706b4d9e
I20250115 14:52:18.395510 847 leader_election.cc:304] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.395735 2549 raft_consensus.cc:2743] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.403568 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1372) took 1402 ms. Trace:
I20250115 14:52:18.403723 1751 rpcz_store.cc:276] 0115 14:52:17.001195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.001245 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:18.119563 (+1118318us) ts_tablet_manager.cc:1145] Deleting tablet c6cb8dfa7c42480bbd139d5acf059a57
0115 14:52:18.160723 (+ 41160us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.320545 (+159822us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.361778 (+ 41233us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.403545 (+ 41767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1118051}
I20250115 14:52:18.404124 1751 tablet_replica.cc:331] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:18.404541 1751 raft_consensus.cc:2238] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.404732 1751 raft_consensus.cc:2267] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.404819 1164 tablet_bootstrap.cc:492] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:18.405339 1751 ts_tablet_manager.cc:1905] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.406193 1164 tablet_bootstrap.cc:654] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.410090 1095 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:52:18.411401 2839 raft_consensus.cc:491] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.411512 2839 raft_consensus.cc:513] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.412056 2839 leader_election.cc:290] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.412578 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7718729182a340fe9b2dfea5a8321c1f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:18.413192 983 leader_election.cc:343] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.414194 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7718729182a340fe9b2dfea5a8321c1f" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.414728 984 leader_election.cc:343] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.414829 984 leader_election.cc:304] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.415351 2839 raft_consensus.cc:2743] T 7718729182a340fe9b2dfea5a8321c1f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.426533 1168 tablet_bootstrap.cc:492] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:18.426748 1168 ts_tablet_manager.cc:1397] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.067s user 0.018s sys 0.005s
I20250115 14:52:18.429354 1168 raft_consensus.cc:357] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.429986 1168 raft_consensus.cc:383] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.430603 1168 raft_consensus.cc:738] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.430881 1168 consensus_queue.cc:260] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.431461 1168 ts_tablet_manager.cc:1428] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250115 14:52:18.431859 1168 tablet_bootstrap.cc:492] T 72a97d102c174bc3a5e3a3c5762033b3 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:18.433223 1168 tablet_bootstrap.cc:654] T 72a97d102c174bc3a5e3a3c5762033b3 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.449000 452 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet a5e97279185541b992340c24ede01656 (table default.Table_a8391687956e4259a71808f267cb4ff0 [id=361ed20953ab4e79b1112ae76b95a423]). Expected version 3 got 0
I20250115 14:52:18.450100 959 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:52:18.451442 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d1c9cfaca7974878b6bf92f9b8cdbbf5" } modify_external_catalogs: true
I20250115 14:52:18.451625 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_d1c9cfaca7974878b6bf92f9b8cdbbf5" } modify_external_catalogs: true
I20250115 14:52:18.452389 2992 sys_catalog.cc:263] Verifying existing consensus state
W20250115 14:52:18.454512 454 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0159e9b3b87844a393d958668e318124 consensus state has no leader
I20250115 14:52:18.454882 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_b2f69c5cd4584eaaa57e55aebd055324"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.455421 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2f69c5cd4584eaaa57e55aebd055324 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.465705 451 catalog_manager.cc:6071] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: default.Table_1426afacf12444b5a8f8977fbac29d75 [id=f6e668ec62be4a1dbd638f0d323b114a] alter complete (version 2)
I20250115 14:52:18.465888 1164 tablet_bootstrap.cc:492] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:18.466037 1164 ts_tablet_manager.cc:1397] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.008s
I20250115 14:52:18.466787 1164 raft_consensus.cc:357] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.467190 1164 raft_consensus.cc:383] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.467257 1164 raft_consensus.cc:738] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.467440 1164 consensus_queue.cc:260] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.472344 451 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_799a6df4f73a45c9a57ced3b6a102091" } modify_external_catalogs: true
I20250115 14:52:18.472504 451 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_799a6df4f73a45c9a57ced3b6a102091" } modify_external_catalogs: true
I20250115 14:52:18.473922 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_164c44d468294cb695fb0a3e317a2e69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:18.474202 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e4c2707915e4e97ad6c66cb249c6d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 6afbdd0b585e4e7e82067dce061924d0) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.474440 1166 tablet_replica.cc:331] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
W20250115 14:52:18.474442 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_164c44d468294cb695fb0a3e317a2e69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.474625 1166 raft_consensus.cc:2238] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.474768 1166 raft_consensus.cc:2267] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.475881 1166 ts_tablet_manager.cc:1905] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.476040 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_599e119acc2046c3bf5cde76e409e873"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.476514 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_599e119acc2046c3bf5cde76e409e873 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.486986 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:52:18.498323 1167 tablet_replica.cc:331] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:18.498515 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f541e916d264738842cd965c7a2a5ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.498944 1167 raft_consensus.cc:2238] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.499173 1167 raft_consensus.cc:2267] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.501385 1166 ts_tablet_manager.cc:1918] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.501498 1166 log.cc:1198] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/2e4c2707915e4e97ad6c66cb249c6d9e
I20250115 14:52:18.501775 1166 ts_tablet_manager.cc:1939] T 2e4c2707915e4e97ad6c66cb249c6d9e P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:18.502120 1167 ts_tablet_manager.cc:1905] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.502116 1751 ts_tablet_manager.cc:1918] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250115 14:52:18.502202 447 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be82f0894ca240e6954fc7af3c41ce23 has no consensus state
I20250115 14:52:18.502351 1751 log.cc:1198] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/57aecd22f97c41618f90d14c2f53a7c9
I20250115 14:52:18.502703 1751 ts_tablet_manager.cc:1939] T 57aecd22f97c41618f90d14c2f53a7c9 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:18.502673 877 tablet_service.cc:1514] Processing DeleteTablet for tablet a48a0af9422f43cf86531665c0fa1aee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.503472 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1373) took 1490 ms. Trace:
I20250115 14:52:18.504168 447 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 4655600323f944969cfc4eaf5bbbb684 (table default.Table_8545a98d01ad4896aefc8375c21bef6f [id=d0eba937a20844a484d0bdc69ef71524]). Expected version 2 got 0
W20250115 14:52:18.504510 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f3c12a0a7eb42c5888961103f7eb178 consensus state has no leader
I20250115 14:52:18.504768 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_8db4c9b252ac4eaf90ab6b556fdbdefc" } new_table_name: "default.Table_0c279fea912144cdb98dd6a75204a207" modify_external_catalogs: true
I20250115 14:52:18.506925 2992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc2e23c560f4096be7b14006bdfc525: Bootstrap starting.
I20250115 14:52:18.507519 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_888f5c5df23b4835a34d43300e9f3806" } modify_external_catalogs: true
I20250115 14:52:18.507606 1167 ts_tablet_manager.cc:1918] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.507687 1167 log.cc:1198] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/4f89ec0a5d68408c897f43ab39df62b1
I20250115 14:52:18.507652 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_888f5c5df23b4835a34d43300e9f3806" } modify_external_catalogs: true
I20250115 14:52:18.507963 1167 ts_tablet_manager.cc:1939] T 4f89ec0a5d68408c897f43ab39df62b1 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:18.508215 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1369) took 1492 ms. Trace:
I20250115 14:52:18.508235 1164 ts_tablet_manager.cc:1428] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.042s user 0.001s sys 0.001s
I20250115 14:52:18.510457 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 2e4c2707915e4e97ad6c66cb249c6d9e (table default.Table_eb0e928b8265482990bb5fbf1072985f [id=dfcc6db7be294db4a0f620ab93714f2b]) successfully deleted
I20250115 14:52:18.514724 446 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 04daad461f0b433ba635df87439b5bc1 (table default.Table_c7251196621d49aa966653f5a456c7eb [id=289c1cdb743b49b5893f551afab62d43]). Expected version 1 got 0
I20250115 14:52:18.515326 1164 tablet_bootstrap.cc:492] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:18.517997 1164 tablet_bootstrap.cc:654] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.526499 455 master_service.cc:946] ReplaceTablet: received request to replace tablet 1218f0bdb1e14bd78d76c6a06805f38c from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:18.526966 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 4911a010ccd2462fae7cb940e24f030c
I20250115 14:52:18.527984 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 4911a010ccd2462fae7cb940e24f030c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.528712 1751 rpcz_store.cc:276] 0115 14:52:17.013035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.013128 (+ 93us) service_pool.cc:225] Handling call
0115 14:52:18.404078 (+1390950us) ts_tablet_manager.cc:1145] Deleting tablet 57aecd22f97c41618f90d14c2f53a7c9
0115 14:52:18.404087 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.443871 (+ 39784us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.502015 (+ 58144us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.503452 (+ 1437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1390568}
I20250115 14:52:18.529187 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1376) took 1413 ms. Trace:
I20250115 14:52:18.529217 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 57aecd22f97c41618f90d14c2f53a7c9 (table default.Table_9118acb2bd5e43e9b5d4b0d2379fadfb [id=492ed43384bd463dbfa14c65b58044e3]) successfully deleted
I20250115 14:52:18.529268 1751 rpcz_store.cc:276] 0115 14:52:17.115512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.115559 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:18.529049 (+1413490us) ts_tablet_manager.cc:1145] Deleting tablet e663111811b34a578bc344c59be8649e
0115 14:52:18.529071 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.529170 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1413304}
W20250115 14:52:18.529661 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet e663111811b34a578bc344c59be8649e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e663111811b34a578bc344c59be8649e already in progress: creating tablet
I20250115 14:52:18.527868 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 4911a010ccd2462fae7cb940e24f030c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.516108 1168 tablet_bootstrap.cc:492] T 72a97d102c174bc3a5e3a3c5762033b3 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:18.532208 1168 ts_tablet_manager.cc:1397] T 72a97d102c174bc3a5e3a3c5762033b3 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.012s
W20250115 14:52:18.532697 848 consensus_peers.cc:487] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d -> Peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Couldn't send request to peer d2d7ecece2ab49e1be7079216517d13b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad988e25800f4af9a069b5a80582f749. This is attempt 1: this message will repeat every 5th retry.
I20250115 14:52:18.533042 1168 raft_consensus.cc:357] T 72a97d102c174bc3a5e3a3c5762033b3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.533182 751 tablet_service.cc:1514] Processing DeleteTablet for tablet 4911a010ccd2462fae7cb940e24f030c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.533319 1168 raft_consensus.cc:383] T 72a97d102c174bc3a5e3a3c5762033b3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.533428 1168 raft_consensus.cc:738] T 72a97d102c174bc3a5e3a3c5762033b3 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.533674 1168 consensus_queue.cc:260] T 72a97d102c174bc3a5e3a3c5762033b3 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.534031 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1377) took 1415 ms. Trace:
I20250115 14:52:18.534240 1168 ts_tablet_manager.cc:1428] T 72a97d102c174bc3a5e3a3c5762033b3 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:18.534164 1751 rpcz_store.cc:276] 0115 14:52:17.118434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.118477 (+ 43us) service_pool.cc:225] Handling call
0115 14:52:18.533895 (+1415418us) ts_tablet_manager.cc:1145] Deleting tablet c49eefbfc9ce404eb604fb265c6a64d0
0115 14:52:18.533906 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.534013 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1415263}
I20250115 14:52:18.534479 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1378) took 1369 ms. Trace:
I20250115 14:52:18.508282 1167 rpcz_store.cc:276] 0115 14:52:17.015546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.015673 (+ 127us) service_pool.cc:225] Handling call
0115 14:52:18.498262 (+1482589us) ts_tablet_manager.cc:1145] Deleting tablet 4f89ec0a5d68408c897f43ab39df62b1
0115 14:52:18.498276 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.504978 (+ 6702us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.507533 (+ 2555us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.508200 (+ 667us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1351589}
W20250115 14:52:18.534641 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c49eefbfc9ce404eb604fb265c6a64d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49eefbfc9ce404eb604fb265c6a64d0 already in progress: creating tablet
I20250115 14:52:18.534699 1751 rpcz_store.cc:276] 0115 14:52:17.165149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.165198 (+ 49us) service_pool.cc:225] Handling call
0115 14:52:18.534391 (+1369193us) ts_tablet_manager.cc:1145] Deleting tablet 5e2ffb75f16c4244b795c21f8b89d32a
0115 14:52:18.534398 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.534466 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1369020}
I20250115 14:52:18.534838 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 4f89ec0a5d68408c897f43ab39df62b1 (table default.Table_d1198c17efa04e4ea060b6d89580342a [id=5e381e3344a34b53a930fcdfc8c8fa25]) successfully deleted
I20250115 14:52:18.534889 1168 tablet_bootstrap.cc:492] T d1259256b7744e739581125611abbe67 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:18.534989 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1379) took 1351 ms. Trace:
I20250115 14:52:18.535046 1751 rpcz_store.cc:276] 0115 14:52:17.183964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.184019 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:18.534905 (+1350886us) ts_tablet_manager.cc:1145] Deleting tablet c300667f775e40b5a65dce61f4863cae
0115 14:52:18.534912 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.534977 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1350683}
I20250115 14:52:18.535319 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1380) took 1347 ms. Trace:
W20250115 14:52:18.535403 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 5e2ffb75f16c4244b795c21f8b89d32a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e2ffb75f16c4244b795c21f8b89d32a already in progress: creating tablet
I20250115 14:52:18.535387 1751 rpcz_store.cc:276] 0115 14:52:17.187912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.187963 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:18.535238 (+1347275us) ts_tablet_manager.cc:1145] Deleting tablet eeb5ad85b4524994bba041dfb3fc8e1f
0115 14:52:18.535244 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.535307 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347099}
W20250115 14:52:18.535598 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet c300667f775e40b5a65dce61f4863cae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c300667f775e40b5a65dce61f4863cae already in progress: creating tablet
I20250115 14:52:18.535633 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1381) took 1341 ms. Trace:
I20250115 14:52:18.535689 1751 rpcz_store.cc:276] 0115 14:52:17.193741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.193909 (+ 168us) service_pool.cc:225] Handling call
0115 14:52:18.535557 (+1341648us) ts_tablet_manager.cc:1145] Deleting tablet 828be3fd7b634d23b88aebf4633c41fe
0115 14:52:18.535564 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.535622 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1341487}
I20250115 14:52:18.535892 1751 tablet_replica.cc:331] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:18.536072 1751 raft_consensus.cc:2238] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250115 14:52:18.536118 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet eeb5ad85b4524994bba041dfb3fc8e1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeb5ad85b4524994bba041dfb3fc8e1f already in progress: creating tablet
I20250115 14:52:18.536206 1751 raft_consensus.cc:2267] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.536324 1168 tablet_bootstrap.cc:654] T d1259256b7744e739581125611abbe67 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
W20250115 14:52:18.536336 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 828be3fd7b634d23b88aebf4633c41fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 828be3fd7b634d23b88aebf4633c41fe already in progress: creating tablet
W20250115 14:52:18.536666 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 4911a010ccd2462fae7cb940e24f030c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4911a010ccd2462fae7cb940e24f030c already in progress: creating tablet
I20250115 14:52:18.536710 1751 ts_tablet_manager.cc:1905] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.537734 1167 tablet_replica.cc:331] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:18.537950 1167 raft_consensus.cc:2238] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.538403 1167 raft_consensus.cc:2267] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.538945 1167 ts_tablet_manager.cc:1905] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.548591 769 raft_consensus.cc:3054] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:18.540416 2839 raft_consensus.cc:491] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.549927 2839 raft_consensus.cc:513] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.550598 2839 leader_election.cc:290] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.551306 772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18a88c8f3ce4a24b7e1e13966760364" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.551863 984 leader_election.cc:343] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.552554 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d18a88c8f3ce4a24b7e1e13966760364" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:18.553107 983 leader_election.cc:343] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.553202 983 leader_election.cc:304] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.554512 2839 raft_consensus.cc:2743] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.556504 1167 ts_tablet_manager.cc:1918] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.556609 1167 log.cc:1198] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ded663ac1197454bbde3c4c2d526002d
I20250115 14:52:18.556903 1167 ts_tablet_manager.cc:1939] T ded663ac1197454bbde3c4c2d526002d P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:18.557173 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1364) took 1542 ms. Trace:
I20250115 14:52:18.557260 1167 rpcz_store.cc:276] 0115 14:52:17.014525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.015307 (+ 782us) service_pool.cc:225] Handling call
0115 14:52:18.537686 (+1522379us) ts_tablet_manager.cc:1145] Deleting tablet ded663ac1197454bbde3c4c2d526002d
0115 14:52:18.537695 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.548592 (+ 10897us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.556407 (+ 7815us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.557157 (+ 750us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1521655}
I20250115 14:52:18.557616 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet ded663ac1197454bbde3c4c2d526002d (table default.Table_ca5ad5415b9144e583b4c771d24d4431 [id=a9793eda46fd4d07963603331438decb]) successfully deleted
I20250115 14:52:18.561160 1167 tablet_replica.cc:331] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:18.561357 1167 raft_consensus.cc:2238] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.561501 1167 raft_consensus.cc:2267] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.562026 1167 ts_tablet_manager.cc:1905] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.573086 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet abefc717e0eb41b18725ef0c73154fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.575989 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd94e4a7fef44f5be462e7a77e8a123 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:00 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.576378 1166 tablet_replica.cc:331] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:18.576579 1166 raft_consensus.cc:2238] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.576719 1166 raft_consensus.cc:2267] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.578478 1166 ts_tablet_manager.cc:1905] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.580273 1162 tablet_bootstrap.cc:492] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.580423 1162 ts_tablet_manager.cc:1397] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.230s user 0.017s sys 0.025s
I20250115 14:52:18.581185 1162 raft_consensus.cc:357] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.581365 1162 raft_consensus.cc:383] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.581406 1162 raft_consensus.cc:738] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.581736 1162 consensus_queue.cc:260] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.582224 1162 ts_tablet_manager.cc:1428] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:18.582866 1162 tablet_bootstrap.cc:492] T 87026b6faf75437eac21dedab1f41ed4 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.584533 1162 tablet_bootstrap.cc:654] T 87026b6faf75437eac21dedab1f41ed4 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.592476 1166 ts_tablet_manager.cc:1918] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.592681 1166 log.cc:1198] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/0fd94e4a7fef44f5be462e7a77e8a123
I20250115 14:52:18.593106 1166 ts_tablet_manager.cc:1939] T 0fd94e4a7fef44f5be462e7a77e8a123 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:18.593415 1162 tablet_bootstrap.cc:492] T 87026b6faf75437eac21dedab1f41ed4 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.593562 1162 ts_tablet_manager.cc:1397] T 87026b6faf75437eac21dedab1f41ed4 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20250115 14:52:18.593794 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 0fd94e4a7fef44f5be462e7a77e8a123 (table default.Table_ef39f0505cbe4de9b46458d25eafb9e1 [id=4fe5c41359724041a1bcf5ab772e68a2]) successfully deleted
I20250115 14:52:18.594393 1162 raft_consensus.cc:357] T 87026b6faf75437eac21dedab1f41ed4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.594640 1162 raft_consensus.cc:383] T 87026b6faf75437eac21dedab1f41ed4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.594713 1162 raft_consensus.cc:738] T 87026b6faf75437eac21dedab1f41ed4 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.594908 1162 consensus_queue.cc:260] T 87026b6faf75437eac21dedab1f41ed4 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.595736 451 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 453adba8c772446c9e24f20eed56a951
I20250115 14:52:18.596716 1162 ts_tablet_manager.cc:1428] T 87026b6faf75437eac21dedab1f41ed4 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250115 14:52:18.596987 1162 tablet_bootstrap.cc:492] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.597483 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 453adba8c772446c9e24f20eed56a951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.597548 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 453adba8c772446c9e24f20eed56a951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.598201 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 453adba8c772446c9e24f20eed56a951 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
W20250115 14:52:18.598846 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 453adba8c772446c9e24f20eed56a951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 453adba8c772446c9e24f20eed56a951 already in progress: creating tablet
I20250115 14:52:18.598956 1162 tablet_bootstrap.cc:654] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.599289 2998 tablet.cc:1721] T 1d1df21ab18f45a8adf9fedd22b2b8a9 P d2d7ecece2ab49e1be7079216517d13b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250115 14:52:18.594522 1751 ts_tablet_manager.cc:1918] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.602453 1751 log.cc:1198] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/02ce16fea4f9469c9fc06fa24f6554be
I20250115 14:52:18.602566 1167 ts_tablet_manager.cc:1918] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.602689 1167 log.cc:1198] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/6308f68e41554470b7d1321c60d4eee9
I20250115 14:52:18.602955 1167 ts_tablet_manager.cc:1939] T 6308f68e41554470b7d1321c60d4eee9 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:18.603039 1751 ts_tablet_manager.cc:1939] T 02ce16fea4f9469c9fc06fa24f6554be P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:18.603220 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1363) took 1588 ms. Trace:
I20250115 14:52:18.603291 1167 rpcz_store.cc:276] 0115 14:52:17.014409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.016058 (+ 1649us) service_pool.cc:225] Handling call
0115 14:52:18.561111 (+1545053us) ts_tablet_manager.cc:1145] Deleting tablet 6308f68e41554470b7d1321c60d4eee9
0115 14:52:18.561122 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.578072 (+ 16950us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.602460 (+ 24388us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.603207 (+ 747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1544420}
I20250115 14:52:18.604672 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 6308f68e41554470b7d1321c60d4eee9 (table default.Table_33f4705d7a1b474d82734fe4a80c0424 [id=b0ebc66f46364e51b73bdee697f00b62]) successfully deleted
I20250115 14:52:18.612740 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_213f670682f841518bdfae30c8c19d67" } modify_external_catalogs: true
I20250115 14:52:18.612915 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_213f670682f841518bdfae30c8c19d67" } modify_external_catalogs: true
I20250115 14:52:18.613754 449 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_254b5a8c44db4a788415c8b69471bf9b" } modify_external_catalogs: true
I20250115 14:52:18.613896 449 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_254b5a8c44db4a788415c8b69471bf9b" } modify_external_catalogs: true
I20250115 14:52:18.614403 1167 tablet_replica.cc:331] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:18.614598 1167 raft_consensus.cc:2238] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.614754 1167 raft_consensus.cc:2267] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.616747 1167 ts_tablet_manager.cc:1905] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.618719 451 catalog_manager.cc:5618] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127) does not have the latest schema for tablet 04daad461f0b433ba635df87439b5bc1 (table default.Table_c7251196621d49aa966653f5a456c7eb [id=289c1cdb743b49b5893f551afab62d43]). Expected version 1 got 0
I20250115 14:52:18.619539 451 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_6472f60e6915482395c4388061880777" } new_table_name: "default.Table_06b768a6fe7648e6b48678522dd0c6a4" modify_external_catalogs: true
I20250115 14:52:18.621366 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 72a97d102c174bc3a5e3a3c5762033b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:02 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.621507 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_376ce4279a6247dca04c0cdca7f71356"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.621995 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_376ce4279a6247dca04c0cdca7f71356 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.626084 446 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fe9050919e854c0b8de275e8a6469dbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.626647 446 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe9050919e854c0b8de275e8a6469dbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.626955 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1382) took 1366 ms. Trace:
I20250115 14:52:18.627108 1751 rpcz_store.cc:276] 0115 14:52:17.260672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.260723 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:18.535857 (+1275134us) ts_tablet_manager.cc:1145] Deleting tablet 02ce16fea4f9469c9fc06fa24f6554be
0115 14:52:18.535865 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.553916 (+ 18051us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.594412 (+ 40496us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.626933 (+ 32521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1274936}
I20250115 14:52:18.627593 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 02ce16fea4f9469c9fc06fa24f6554be (table default.Table_a2eb9b62487c4908880b44b9915e4cf7 [id=d15ba63620364fd18f1751a535cc3235]) successfully deleted
I20250115 14:52:18.627935 1751 tablet_replica.cc:331] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:18.628214 1751 raft_consensus.cc:2238] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.628474 1751 raft_consensus.cc:2267] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250115 14:52:18.628877 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1512a0e99354e7096d4da71dcb95e02 consensus state has no leader
I20250115 14:52:18.629192 1751 ts_tablet_manager.cc:1905] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.629858 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 5f09898e70cd4ef2b05ea3f567a3137d
I20250115 14:52:18.610409 1162 tablet_bootstrap.cc:492] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.630016 1162 ts_tablet_manager.cc:1397] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20250115 14:52:18.631142 1162 raft_consensus.cc:357] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.631325 1162 raft_consensus.cc:383] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.631369 1162 raft_consensus.cc:738] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.631662 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f09898e70cd4ef2b05ea3f567a3137d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.631707 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f09898e70cd4ef2b05ea3f567a3137d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.631947 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f09898e70cd4ef2b05ea3f567a3137d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:18.632344 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 5f09898e70cd4ef2b05ea3f567a3137d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f09898e70cd4ef2b05ea3f567a3137d already in progress: creating tablet
I20250115 14:52:18.634573 1162 consensus_queue.cc:260] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.635147 1168 tablet_bootstrap.cc:492] T d1259256b7744e739581125611abbe67 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:18.635316 1168 ts_tablet_manager.cc:1397] T d1259256b7744e739581125611abbe67 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.100s user 0.013s sys 0.005s
I20250115 14:52:18.636013 1168 raft_consensus.cc:357] T d1259256b7744e739581125611abbe67 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.636242 1168 raft_consensus.cc:383] T d1259256b7744e739581125611abbe67 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.636428 1168 raft_consensus.cc:738] T d1259256b7744e739581125611abbe67 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.636693 1168 consensus_queue.cc:260] T d1259256b7744e739581125611abbe67 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.637240 1168 ts_tablet_manager.cc:1428] T d1259256b7744e739581125611abbe67 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250115 14:52:18.637532 1168 tablet_bootstrap.cc:492] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:18.638882 1168 tablet_bootstrap.cc:654] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.639644 1162 ts_tablet_manager.cc:1428] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250115 14:52:18.639919 1162 tablet_bootstrap.cc:492] T fab76e0427f14da48bf101db57f0457c P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.641500 1162 tablet_bootstrap.cc:654] T fab76e0427f14da48bf101db57f0457c P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.649847 1167 ts_tablet_manager.cc:1918] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.652886 1167 log.cc:1198] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/398ac8af619c427287b3964443173864
I20250115 14:52:18.654448 2549 raft_consensus.cc:491] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.656849 2549 raft_consensus.cc:513] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.657426 2549 leader_election.cc:290] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:18.657728 1167 ts_tablet_manager.cc:1939] T 398ac8af619c427287b3964443173864 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:18.658025 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1365) took 1643 ms. Trace:
I20250115 14:52:18.658098 1167 rpcz_store.cc:276] 0115 14:52:17.014618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.016721 (+ 2103us) service_pool.cc:225] Handling call
0115 14:52:18.614340 (+1597619us) ts_tablet_manager.cc:1145] Deleting tablet 398ac8af619c427287b3964443173864
0115 14:52:18.614352 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.619634 (+ 5282us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.649720 (+ 30086us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.658011 (+ 8291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597432}
I20250115 14:52:18.658159 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f541e916d264738842cd965c7a2a5ec" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.658818 848 leader_election.cc:343] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.659690 1168 tablet_bootstrap.cc:492] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:18.659880 1168 ts_tablet_manager.cc:1397] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250115 14:52:18.660617 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 398ac8af619c427287b3964443173864 (table default.Table_a26e212086b1478cb159097102934dd1 [id=4bb94d3cca0c4a14aad6957911139e51]) successfully deleted
W20250115 14:52:18.660691 847 leader_election.cc:343] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: 3f541e916d264738842cd965c7a2a5ec
I20250115 14:52:18.660782 847 leader_election.cc:304] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.661103 2549 raft_consensus.cc:2743] T 3f541e916d264738842cd965c7a2a5ec P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:18.662595 451 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f38727d283924a14a5915a5010128d6e consensus state has no leader
I20250115 14:52:18.663619 2839 raft_consensus.cc:491] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.663765 2839 raft_consensus.cc:513] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.664426 2839 leader_election.cc:290] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.664933 449 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 806bf29f5dca4c55bdb14193e8fa2a82
I20250115 14:52:18.665195 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5882252383ba46aba0ec053fab1fd383" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:18.665740 983 leader_election.cc:343] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.666232 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 806bf29f5dca4c55bdb14193e8fa2a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.666293 743 tablet_service.cc:1514] Processing DeleteTablet for tablet 806bf29f5dca4c55bdb14193e8fa2a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.667845 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5882252383ba46aba0ec053fab1fd383" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.668515 984 leader_election.cc:343] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.668627 984 leader_election.cc:304] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:18.668895 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 806bf29f5dca4c55bdb14193e8fa2a82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 806bf29f5dca4c55bdb14193e8fa2a82 already in progress: creating tablet
I20250115 14:52:18.669198 2915 raft_consensus.cc:2743] T 5882252383ba46aba0ec053fab1fd383 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.671422 1017 tablet_service.cc:1514] Processing DeleteTablet for tablet 806bf29f5dca4c55bdb14193e8fa2a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.671847 1168 raft_consensus.cc:357] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.672089 1168 raft_consensus.cc:383] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.672194 1168 raft_consensus.cc:738] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.672474 1168 consensus_queue.cc:260] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.673059 1168 ts_tablet_manager.cc:1428] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250115 14:52:18.673373 1168 tablet_bootstrap.cc:492] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:18.677410 1168 tablet_bootstrap.cc:654] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.677727 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1370) took 1565 ms. Trace:
I20250115 14:52:18.682024 1167 rpcz_store.cc:276] 0115 14:52:17.112562 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.112613 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:18.677612 (+1564999us) ts_tablet_manager.cc:1145] Deleting tablet e663111811b34a578bc344c59be8649e
0115 14:52:18.677621 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.677712 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564842}
I20250115 14:52:18.682623 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1371) took 1562 ms. Trace:
I20250115 14:52:18.680327 959 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:52:18.682762 1167 rpcz_store.cc:276] 0115 14:52:17.119990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.120262 (+ 272us) service_pool.cc:225] Handling call
0115 14:52:18.682515 (+1562253us) ts_tablet_manager.cc:1145] Deleting tablet c49eefbfc9ce404eb604fb265c6a64d0
0115 14:52:18.682524 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.682608 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562046}
I20250115 14:52:18.683943 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1373) took 1518 ms. Trace:
I20250115 14:52:18.684070 1167 rpcz_store.cc:276] 0115 14:52:17.165087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.165139 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:18.683237 (+1518098us) ts_tablet_manager.cc:1145] Deleting tablet 5e2ffb75f16c4244b795c21f8b89d32a
0115 14:52:18.683846 (+ 609us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.683926 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1517916}
I20250115 14:52:18.688179 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1374) took 1499 ms. Trace:
W20250115 14:52:18.682520 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet e663111811b34a578bc344c59be8649e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e663111811b34a578bc344c59be8649e already in progress: creating tablet
W20250115 14:52:18.689395 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c49eefbfc9ce404eb604fb265c6a64d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c49eefbfc9ce404eb604fb265c6a64d0 already in progress: creating tablet
W20250115 14:52:18.689646 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 5e2ffb75f16c4244b795c21f8b89d32a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e2ffb75f16c4244b795c21f8b89d32a already in progress: creating tablet
I20250115 14:52:18.693871 1167 rpcz_store.cc:276] 0115 14:52:17.188455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.188639 (+ 184us) service_pool.cc:225] Handling call
0115 14:52:18.688059 (+1499420us) ts_tablet_manager.cc:1145] Deleting tablet c300667f775e40b5a65dce61f4863cae
0115 14:52:18.688068 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.688159 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1499231}
W20250115 14:52:18.694306 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet c300667f775e40b5a65dce61f4863cae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c300667f775e40b5a65dce61f4863cae already in progress: creating tablet
I20250115 14:52:18.694392 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1375) took 1505 ms. Trace:
I20250115 14:52:18.694454 1167 rpcz_store.cc:276] 0115 14:52:17.188592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.188859 (+ 267us) service_pool.cc:225] Handling call
0115 14:52:18.694283 (+1505424us) ts_tablet_manager.cc:1145] Deleting tablet eeb5ad85b4524994bba041dfb3fc8e1f
0115 14:52:18.694294 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.694376 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1505308}
W20250115 14:52:18.695117 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet eeb5ad85b4524994bba041dfb3fc8e1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeb5ad85b4524994bba041dfb3fc8e1f already in progress: creating tablet
I20250115 14:52:18.696552 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1376) took 1502 ms. Trace:
I20250115 14:52:18.696630 1167 rpcz_store.cc:276] 0115 14:52:17.193661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.193708 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:18.696445 (+1502737us) ts_tablet_manager.cc:1145] Deleting tablet 828be3fd7b634d23b88aebf4633c41fe
0115 14:52:18.696456 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.696538 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502571}
W20250115 14:52:18.697049 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 828be3fd7b634d23b88aebf4633c41fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 828be3fd7b634d23b88aebf4633c41fe already in progress: creating tablet
I20250115 14:52:18.697099 1167 tablet_replica.cc:331] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:18.697278 1167 raft_consensus.cc:2238] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.697438 1167 raft_consensus.cc:2267] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.697983 1167 ts_tablet_manager.cc:1905] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.702373 1168 tablet_bootstrap.cc:492] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:18.702517 1168 ts_tablet_manager.cc:1397] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250115 14:52:18.703250 1168 raft_consensus.cc:357] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.703456 1168 raft_consensus.cc:383] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.703502 1168 raft_consensus.cc:738] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.703717 1168 consensus_queue.cc:260] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.704242 1168 ts_tablet_manager.cc:1428] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:18.708580 1168 tablet_bootstrap.cc:492] T d6f2bc62621d402cacfcbe2eb74173f9 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:18.710918 1168 tablet_bootstrap.cc:654] T d6f2bc62621d402cacfcbe2eb74173f9 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.724937 1751 ts_tablet_manager.cc:1918] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.725080 1751 log.cc:1198] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/342bf8d4dc4244f982b1b63aaae0fb45
I20250115 14:52:18.725350 1751 ts_tablet_manager.cc:1939] T 342bf8d4dc4244f982b1b63aaae0fb45 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:18.726070 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1387) took 1435 ms. Trace:
I20250115 14:52:18.726168 1751 rpcz_store.cc:276] 0115 14:52:17.290282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.290344 (+ 62us) service_pool.cc:225] Handling call
0115 14:52:18.627885 (+1337541us) ts_tablet_manager.cc:1145] Deleting tablet 342bf8d4dc4244f982b1b63aaae0fb45
0115 14:52:18.627896 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.654897 (+ 27001us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.724821 (+ 69924us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.726049 (+ 1228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337347}
I20250115 14:52:18.726604 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1394) took 1392 ms. Trace:
I20250115 14:52:18.726676 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 342bf8d4dc4244f982b1b63aaae0fb45 (table default.Table_1ee3b83d4fee4651b3979b44d4903db6 [id=1822bed451a74f2dbd54da029abd4384]) successfully deleted
I20250115 14:52:18.726675 1751 rpcz_store.cc:276] 0115 14:52:17.334556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.334609 (+ 53us) service_pool.cc:225] Handling call
0115 14:52:18.726487 (+1391878us) ts_tablet_manager.cc:1145] Deleting tablet 126e01e9520544619f3a124586937fb5
0115 14:52:18.726496 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.726590 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1391692}
I20250115 14:52:18.726969 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1395) took 1392 ms. Trace:
I20250115 14:52:18.727032 1751 rpcz_store.cc:276] 0115 14:52:17.334681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.334728 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:18.726881 (+1392153us) ts_tablet_manager.cc:1145] Deleting tablet 7b6ea13ef1884d66bd1e7655df3e925d
0115 14:52:18.726889 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.726956 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1392034}
I20250115 14:52:18.727288 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1406) took 1178 ms. Trace:
I20250115 14:52:18.727351 1751 rpcz_store.cc:276] 0115 14:52:17.548730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.548811 (+ 81us) service_pool.cc:225] Handling call
0115 14:52:18.727205 (+1178394us) ts_tablet_manager.cc:1145] Deleting tablet bc06a77162a6430bbbda7a3ca2c4e9ab
0115 14:52:18.727213 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.727274 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1178194}
W20250115 14:52:18.727408 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 126e01e9520544619f3a124586937fb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 126e01e9520544619f3a124586937fb5 already in progress: creating tablet
I20250115 14:52:18.727605 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1408) took 1166 ms. Trace:
W20250115 14:52:18.727608 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 7b6ea13ef1884d66bd1e7655df3e925d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b6ea13ef1884d66bd1e7655df3e925d already in progress: creating tablet
I20250115 14:52:18.727663 1751 rpcz_store.cc:276] 0115 14:52:17.561311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.561359 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:18.727527 (+1166168us) ts_tablet_manager.cc:1145] Deleting tablet 417c5441bb7147a98b8ce0cd823a1855
0115 14:52:18.727535 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.727594 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1166029}
I20250115 14:52:18.727943 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1409) took 1165 ms. Trace:
I20250115 14:52:18.728009 1751 rpcz_store.cc:276] 0115 14:52:17.562020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.562068 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:18.727860 (+1165792us) ts_tablet_manager.cc:1145] Deleting tablet df0d072330444439a6b9b87db099c8b1
0115 14:52:18.727868 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.727931 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1165663}
W20250115 14:52:18.728672 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet bc06a77162a6430bbbda7a3ca2c4e9ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc06a77162a6430bbbda7a3ca2c4e9ab already in progress: creating tablet
W20250115 14:52:18.728888 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 417c5441bb7147a98b8ce0cd823a1855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417c5441bb7147a98b8ce0cd823a1855 already in progress: creating tablet
W20250115 14:52:18.729096 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet df0d072330444439a6b9b87db099c8b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df0d072330444439a6b9b87db099c8b1 already in progress: creating tablet
I20250115 14:52:18.729863 1164 tablet_bootstrap.cc:492] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:18.730059 1164 ts_tablet_manager.cc:1397] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.215s user 0.022s sys 0.019s
I20250115 14:52:18.730842 1164 raft_consensus.cc:357] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.731096 1164 raft_consensus.cc:383] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.731191 1164 raft_consensus.cc:738] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.731448 1164 consensus_queue.cc:260] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.735666 1167 ts_tablet_manager.cc:1918] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.735809 1167 log.cc:1198] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/48b15de9bf8943cdb9e5abd943484993
I20250115 14:52:18.737074 1164 ts_tablet_manager.cc:1428] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250115 14:52:18.736100 1167 ts_tablet_manager.cc:1939] T 48b15de9bf8943cdb9e5abd943484993 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:18.737862 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1377) took 1515 ms. Trace:
I20250115 14:52:18.737942 1167 rpcz_store.cc:276] 0115 14:52:17.222682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.222730 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:18.697054 (+1474324us) ts_tablet_manager.cc:1145] Deleting tablet 48b15de9bf8943cdb9e5abd943484993
0115 14:52:18.697063 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.715738 (+ 18675us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.735542 (+ 19804us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.737843 (+ 2301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474154}
I20250115 14:52:18.738830 1168 tablet_bootstrap.cc:492] T d6f2bc62621d402cacfcbe2eb74173f9 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:18.738951 1168 ts_tablet_manager.cc:1397] T d6f2bc62621d402cacfcbe2eb74173f9 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.002s
I20250115 14:52:18.739604 1168 raft_consensus.cc:357] T d6f2bc62621d402cacfcbe2eb74173f9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.739802 1168 raft_consensus.cc:383] T d6f2bc62621d402cacfcbe2eb74173f9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.739854 1168 raft_consensus.cc:738] T d6f2bc62621d402cacfcbe2eb74173f9 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.740064 1168 consensus_queue.cc:260] T d6f2bc62621d402cacfcbe2eb74173f9 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.740576 1168 ts_tablet_manager.cc:1428] T d6f2bc62621d402cacfcbe2eb74173f9 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:18.740825 1168 tablet_bootstrap.cc:492] T ed777d7496514ec5986a7e6f53d532b1 P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:18.741246 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 48b15de9bf8943cdb9e5abd943484993 (table default.Table_a7e733ef2d844d238685b7d25b53c151 [id=c40fb10d4df34d58a790b3d4e39b48e5]) successfully deleted
I20250115 14:52:18.742373 1168 tablet_bootstrap.cc:654] T ed777d7496514ec5986a7e6f53d532b1 P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.742475 1164 tablet_bootstrap.cc:492] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:18.744031 1164 tablet_bootstrap.cc:654] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.744812 447 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a01859d3389e427583c1d017d358e254"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:18.745574 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_4f00ef2ef63d4f77929a245aa9bff035" } new_table_name: "default.Table_494518ca300744e9821f893256ddf6a9" modify_external_catalogs: true
I20250115 14:52:18.746395 449 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2ef4d39c78514fb795a21054fb20be91" } new_table_name: "default.Table_e0e39792bb0e488cbf796a10004f805e" modify_external_catalogs: true
I20250115 14:52:18.746783 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fe98a52b5c2a49fd944bfb33bf2fe450"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.747184 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe98a52b5c2a49fd944bfb33bf2fe450 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.747946 877 tablet_service.cc:1514] Processing DeleteTablet for tablet d1259256b7744e739581125611abbe67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:05 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.748234 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_fb06386c0f2b416e9cbf243ff9c9e00f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.748687 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb06386c0f2b416e9cbf243ff9c9e00f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.750003 1162 tablet_bootstrap.cc:492] T fab76e0427f14da48bf101db57f0457c P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.750129 1162 ts_tablet_manager.cc:1397] T fab76e0427f14da48bf101db57f0457c P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.000s
I20250115 14:52:18.750847 1162 raft_consensus.cc:357] T fab76e0427f14da48bf101db57f0457c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:18.751026 1162 raft_consensus.cc:383] T fab76e0427f14da48bf101db57f0457c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.751080 1162 raft_consensus.cc:738] T fab76e0427f14da48bf101db57f0457c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.751168 446 master_service.cc:946] ReplaceTablet: received request to replace tablet 38e1b2ec1cb84b2f8b2d866f9f30296c from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:18.751262 1162 consensus_queue.cc:260] T fab76e0427f14da48bf101db57f0457c P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:18.751470 452 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_ebb552ba0a2d48fd9270ab5f8bf89f06" } new_table_name: "default.Table_094113b73fd74f17b950bad712b47c7d" modify_external_catalogs: true
I20250115 14:52:18.751713 1162 ts_tablet_manager.cc:1428] T fab76e0427f14da48bf101db57f0457c P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250115 14:52:18.753067 447 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a01859d3389e427583c1d017d358e254 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.754158 1162 tablet_bootstrap.cc:492] T be8e12ccb8d14c1c9258dc3f81d1527d P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.755681 1162 tablet_bootstrap.cc:654] T be8e12ccb8d14c1c9258dc3f81d1527d P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.755955 1168 tablet_bootstrap.cc:492] T ed777d7496514ec5986a7e6f53d532b1 P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:18.756076 1168 ts_tablet_manager.cc:1397] T ed777d7496514ec5986a7e6f53d532b1 P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20250115 14:52:18.756759 1168 raft_consensus.cc:357] T ed777d7496514ec5986a7e6f53d532b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.756947 1168 raft_consensus.cc:383] T ed777d7496514ec5986a7e6f53d532b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.756996 1168 raft_consensus.cc:738] T ed777d7496514ec5986a7e6f53d532b1 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.757190 1168 consensus_queue.cc:260] T ed777d7496514ec5986a7e6f53d532b1 P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.757761 1168 ts_tablet_manager.cc:1428] T ed777d7496514ec5986a7e6f53d532b1 P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250115 14:52:18.758021 1168 tablet_bootstrap.cc:492] T 1218f0bdb1e14bd78d76c6a06805f38c P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:18.760097 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1390) took 1424 ms. Trace:
I20250115 14:52:18.760304 1167 rpcz_store.cc:276] 0115 14:52:17.335851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.357408 (+ 21557us) service_pool.cc:225] Handling call
0115 14:52:18.759961 (+1402553us) ts_tablet_manager.cc:1145] Deleting tablet 7b6ea13ef1884d66bd1e7655df3e925d
0115 14:52:18.759971 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.760079 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1402382}
I20250115 14:52:18.760860 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1389) took 1425 ms. Trace:
W20250115 14:52:18.763345 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 7b6ea13ef1884d66bd1e7655df3e925d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b6ea13ef1884d66bd1e7655df3e925d already in progress: creating tablet
I20250115 14:52:18.763916 1167 rpcz_store.cc:276] 0115 14:52:17.335720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.361792 (+ 26072us) service_pool.cc:225] Handling call
0115 14:52:18.760630 (+1398838us) ts_tablet_manager.cc:1145] Deleting tablet 126e01e9520544619f3a124586937fb5
0115 14:52:18.760758 (+ 128us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.760845 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1398534}
I20250115 14:52:18.765183 1167 tablet_replica.cc:331] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:18.765357 1167 raft_consensus.cc:2238] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.765489 1167 raft_consensus.cc:2267] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.765971 1167 ts_tablet_manager.cc:1905] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250115 14:52:18.766069 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 126e01e9520544619f3a124586937fb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 126e01e9520544619f3a124586937fb5 already in progress: creating tablet
I20250115 14:52:18.769264 2839 raft_consensus.cc:491] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.769402 2839 raft_consensus.cc:513] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.769987 2839 leader_election.cc:290] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.770496 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1218f0bdb1e14bd78d76c6a06805f38c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:18.770947 448 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 866454761ddb4455b7deb0231d413bc6 has no consensus state
W20250115 14:52:18.771008 983 leader_election.cc:343] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.774538 453 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 5882252383ba46aba0ec053fab1fd383 (table default.Table_049f35b1737c479bbff44ef1afe050e2 [id=1532f36247a340cbba3c9b792d500c04]). Expected version 2 got 0
I20250115 14:52:18.775308 453 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_c04871870a4c4ea9b78070b04079345c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.775784 453 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c04871870a4c4ea9b78070b04079345c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.776082 2549 raft_consensus.cc:491] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.776196 2549 raft_consensus.cc:513] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.776830 2549 leader_election.cc:290] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:18.777012 877 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9ce226d3ba44d6908257bbd0df3a12 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.728262 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1410) took 1165 ms. Trace:
I20250115 14:52:18.777252 1751 rpcz_store.cc:276] 0115 14:52:17.562647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.562691 (+ 44us) service_pool.cc:225] Handling call
0115 14:52:18.728185 (+1165494us) ts_tablet_manager.cc:1145] Deleting tablet 87b28e83081a4f6e959c8b26c68b81ce
0115 14:52:18.728192 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.728251 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1165355}
W20250115 14:52:18.777921 440 catalog_manager.cc:4852] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): delete failed for tablet 87b28e83081a4f6e959c8b26c68b81ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87b28e83081a4f6e959c8b26c68b81ce already in progress: creating tablet
W20250115 14:52:18.778074 847 leader_election.cc:343] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: cc9ce226d3ba44d6908257bbd0df3a12
I20250115 14:52:18.778205 1751 tablet_replica.cc:331] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:18.778752 1751 raft_consensus.cc:2238] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.779008 1751 raft_consensus.cc:2267] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.779176 877 tablet_service.cc:1514] Processing DeleteTablet for tablet abefc717e0eb41b18725ef0c73154fdd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:14 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.780055 877 tablet_service.cc:1467] Processing CreateTablet for tablet 68cea2dcdc4f4efb9aff307b46df7502 (DEFAULT_TABLE table=default.Table_fc7d1caa01d349dcb4ffef0c5d7135f7 [id=1ddb732d22634ab9a827fe9c9e55bd40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.780339 1751 ts_tablet_manager.cc:1905] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.780529 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cea2dcdc4f4efb9aff307b46df7502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.780911 2915 raft_consensus.cc:491] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.781018 2915 raft_consensus.cc:513] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.781549 2915 leader_election.cc:290] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.782334 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be094dcd3cb4f9d9259d354300c1600" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:18.782811 983 leader_election.cc:343] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.784544 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 68cea2dcdc4f4efb9aff307b46df7502 (DEFAULT_TABLE table=default.Table_fc7d1caa01d349dcb4ffef0c5d7135f7 [id=1ddb732d22634ab9a827fe9c9e55bd40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.784914 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cea2dcdc4f4efb9aff307b46df7502. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.788158 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed4975796a547f1a8b84565712befde (DEFAULT_TABLE table=default.Table_cca21d06682c49a3a8f1aa144af69dc0 [id=6de10c0e37f74a8da68fef5845ee8963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.788479 1162 tablet_bootstrap.cc:492] T be8e12ccb8d14c1c9258dc3f81d1527d P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.788601 1162 ts_tablet_manager.cc:1397] T be8e12ccb8d14c1c9258dc3f81d1527d P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250115 14:52:18.788640 1010 tablet_service.cc:1467] Processing CreateTablet for tablet be82f0894ca240e6954fc7af3c41ce23 (DEFAULT_TABLE table=default.Table_ebb552ba0a2d48fd9270ab5f8bf89f06 [id=30e48b9b0f2e4d40b628e8309185f91b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.788990 1010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be82f0894ca240e6954fc7af3c41ce23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.789361 1162 raft_consensus.cc:357] T be8e12ccb8d14c1c9258dc3f81d1527d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:18.789547 1162 raft_consensus.cc:383] T be8e12ccb8d14c1c9258dc3f81d1527d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.789593 1162 raft_consensus.cc:738] T be8e12ccb8d14c1c9258dc3f81d1527d P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.789762 1162 consensus_queue.cc:260] T be8e12ccb8d14c1c9258dc3f81d1527d P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:18.789901 1011 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2c81edb81a474a9db7008ff12f2ff0 (DEFAULT_TABLE table=default.Table_7b8c967763964a01a95d0d86f202070c [id=2b2d6d6dab8844ce904edbcd9beb7d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.790184 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2c81edb81a474a9db7008ff12f2ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.790238 1162 ts_tablet_manager.cc:1428] T be8e12ccb8d14c1c9258dc3f81d1527d P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250115 14:52:18.790880 1162 tablet_bootstrap.cc:492] T b47502309761449dae3be07ea6363238 P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.791080 1013 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea716ad60914d848a450eea082e696a (DEFAULT_TABLE table=default.Table_24972c0535bf41ec9c4d4b8e269b2a41 [id=1df25578d5584828b955627842f78ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.791386 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 24e94b77956d4b62862c52a739e44aef (DEFAULT_TABLE table=default.Table_f586057adf44494596993a543597c699 [id=fdf1f810449c4a00b1f118ff4257220e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.791576 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea716ad60914d848a450eea082e696a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:18.796670 452 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be82f0894ca240e6954fc7af3c41ce23 consensus state has no leader
I20250115 14:52:18.797181 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_1fa85362ddff4bde9f0a54134195d723"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.797748 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fa85362ddff4bde9f0a54134195d723 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.799110 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e5e8956be33c4ba7ab1e001e9719a2ef" } modify_external_catalogs: true
I20250115 14:52:18.799250 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e5e8956be33c4ba7ab1e001e9719a2ef" } modify_external_catalogs: true
I20250115 14:52:18.799575 1162 tablet_bootstrap.cc:654] T b47502309761449dae3be07ea6363238 P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.772163 769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1218f0bdb1e14bd78d76c6a06805f38c" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:18.800457 449 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8589265cc054407b68055bf5665057e consensus state has no leader
I20250115 14:52:18.778646 743 tablet_service.cc:1514] Processing DeleteTablet for tablet a300a76b8f7942dea648a20cbb3d5368 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:12 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.778684 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9ce226d3ba44d6908257bbd0df3a12" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:18.800683 449 master_service.cc:946] ReplaceTablet: received request to replace tablet 7cf17435b88c4e87800daefcd4fa35ec from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:18.800876 1166 tablet_replica.cc:331] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:18.801087 1166 raft_consensus.cc:2238] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.801246 1166 raft_consensus.cc:2267] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.783427 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3be094dcd3cb4f9d9259d354300c1600" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:18.801242 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_30525df52f6d4d049b63d5cb20ce28b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:18.785844 751 tablet_service.cc:1467] Processing CreateTablet for tablet 68cea2dcdc4f4efb9aff307b46df7502 (DEFAULT_TABLE table=default.Table_fc7d1caa01d349dcb4ffef0c5d7135f7 [id=1ddb732d22634ab9a827fe9c9e55bd40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.801671 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68cea2dcdc4f4efb9aff307b46df7502. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:18.801704 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30525df52f6d4d049b63d5cb20ce28b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:18.802039 984 leader_election.cc:343] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.802150 984 leader_election.cc:304] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
W20250115 14:52:18.802531 984 leader_election.cc:343] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.802608 984 leader_election.cc:304] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.788489 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 1423ccaf7dfc423ebba097f8f8585e56 (DEFAULT_TABLE table=default.Table_2b33069ab0bb4cfa9d194d732eb8aa8a [id=2c5576e315cd49e4937b3eeb752bc7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.802896 2839 raft_consensus.cc:2743] T 1218f0bdb1e14bd78d76c6a06805f38c P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.803020 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1423ccaf7dfc423ebba097f8f8585e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.803048 2839 raft_consensus.cc:2743] T 3be094dcd3cb4f9d9259d354300c1600 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.790915 1021 tablet_service.cc:1467] Processing CreateTablet for tablet c25912a79e82457c888b6d18a5643d32 (DEFAULT_TABLE table=default.Table_19257b7b80b44471bc2848b06e1e25ee [id=f445659397d9403d90fc5d47e9c463ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.803582 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25912a79e82457c888b6d18a5643d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.803620 1166 ts_tablet_manager.cc:1905] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.804388 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e94b77956d4b62862c52a739e44aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.786502 748 tablet_service.cc:1467] Processing CreateTablet for tablet 1423ccaf7dfc423ebba097f8f8585e56 (DEFAULT_TABLE table=default.Table_2b33069ab0bb4cfa9d194d732eb8aa8a [id=2c5576e315cd49e4937b3eeb752bc7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.805140 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed4975796a547f1a8b84565712befde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.805171 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1423ccaf7dfc423ebba097f8f8585e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.786773 737 tablet_service.cc:1467] Processing CreateTablet for tablet be82f0894ca240e6954fc7af3c41ce23 (DEFAULT_TABLE table=default.Table_ebb552ba0a2d48fd9270ab5f8bf89f06 [id=30e48b9b0f2e4d40b628e8309185f91b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.786917 742 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed4975796a547f1a8b84565712befde (DEFAULT_TABLE table=default.Table_cca21d06682c49a3a8f1aa144af69dc0 [id=6de10c0e37f74a8da68fef5845ee8963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.809527 454 master_service.cc:946] ReplaceTablet: received request to replace tablet 9c35882d2e5c464c9cfad496ba9d6af9 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:18.809582 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be82f0894ca240e6954fc7af3c41ce23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.810091 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed4975796a547f1a8b84565712befde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.786974 747 tablet_service.cc:1467] Processing CreateTablet for tablet c154e7d96dc64070bef1bccc85d9e51a (DEFAULT_TABLE table=default.Table_3ba14975b9434368bb1ad7ce335a7baa [id=9f4c126d1202439692b44a9bedfb2f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.810364 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c154e7d96dc64070bef1bccc85d9e51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.787021 756 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2c81edb81a474a9db7008ff12f2ff0 (DEFAULT_TABLE table=default.Table_7b8c967763964a01a95d0d86f202070c [id=2b2d6d6dab8844ce904edbcd9beb7d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.787158 746 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea716ad60914d848a450eea082e696a (DEFAULT_TABLE table=default.Table_24972c0535bf41ec9c4d4b8e269b2a41 [id=1df25578d5584828b955627842f78ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.787240 749 tablet_service.cc:1467] Processing CreateTablet for tablet c25912a79e82457c888b6d18a5643d32 (DEFAULT_TABLE table=default.Table_19257b7b80b44471bc2848b06e1e25ee [id=f445659397d9403d90fc5d47e9c463ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.787499 750 tablet_service.cc:1467] Processing CreateTablet for tablet 24e94b77956d4b62862c52a739e44aef (DEFAULT_TABLE table=default.Table_f586057adf44494596993a543597c699 [id=fdf1f810449c4a00b1f118ff4257220e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.814574 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2c81edb81a474a9db7008ff12f2ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.814585 2915 raft_consensus.cc:491] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.814692 2915 raft_consensus.cc:513] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.787813 741 tablet_service.cc:1467] Processing CreateTablet for tablet a17189ed67174b9682bab32959b729e3 (DEFAULT_TABLE table=default.Table_7052ed5e2bf04d48b7d75dd702094b65 [id=bedc93977f754793b6eeb68aec944c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.815245 2915 leader_election.cc:290] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.815732 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abefc717e0eb41b18725ef0c73154fdd" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:18.815917 908 raft_consensus.cc:2463] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 0.
I20250115 14:52:18.816258 983 leader_election.cc:304] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters:
W20250115 14:52:18.816640 848 leader_election.cc:343] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.816742 848 leader_election.cc:304] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.816849 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abefc717e0eb41b18725ef0c73154fdd" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:18.816984 3003 raft_consensus.cc:2743] T cc9ce226d3ba44d6908257bbd0df3a12 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250115 14:52:18.817384 984 leader_election.cc:343] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.817804 884 tablet_service.cc:1467] Processing CreateTablet for tablet c25912a79e82457c888b6d18a5643d32 (DEFAULT_TABLE table=default.Table_19257b7b80b44471bc2848b06e1e25ee [id=f445659397d9403d90fc5d47e9c463ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.817827 883 tablet_service.cc:1467] Processing CreateTablet for tablet 1423ccaf7dfc423ebba097f8f8585e56 (DEFAULT_TABLE table=default.Table_2b33069ab0bb4cfa9d194d732eb8aa8a [id=2c5576e315cd49e4937b3eeb752bc7f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.818161 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25912a79e82457c888b6d18a5643d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.818126 889 tablet_service.cc:1467] Processing CreateTablet for tablet 5ed4975796a547f1a8b84565712befde (DEFAULT_TABLE table=default.Table_cca21d06682c49a3a8f1aa144af69dc0 [id=6de10c0e37f74a8da68fef5845ee8963]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.818186 882 tablet_service.cc:1467] Processing CreateTablet for tablet 3ea716ad60914d848a450eea082e696a (DEFAULT_TABLE table=default.Table_24972c0535bf41ec9c4d4b8e269b2a41 [id=1df25578d5584828b955627842f78ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.818445 891 tablet_service.cc:1467] Processing CreateTablet for tablet be82f0894ca240e6954fc7af3c41ce23 (DEFAULT_TABLE table=default.Table_ebb552ba0a2d48fd9270ab5f8bf89f06 [id=30e48b9b0f2e4d40b628e8309185f91b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.818739 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2c81edb81a474a9db7008ff12f2ff0 (DEFAULT_TABLE table=default.Table_7b8c967763964a01a95d0d86f202070c [id=2b2d6d6dab8844ce904edbcd9beb7d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.819002 2915 raft_consensus.cc:2798] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250115 14:52:18.819033 890 tablet_service.cc:1467] Processing CreateTablet for tablet 24e94b77956d4b62862c52a739e44aef (DEFAULT_TABLE table=default.Table_f586057adf44494596993a543597c699 [id=fdf1f810449c4a00b1f118ff4257220e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.819088 2915 raft_consensus.cc:491] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250115 14:52:18.819144 2915 raft_consensus.cc:3054] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:18.819301 893 tablet_service.cc:1467] Processing CreateTablet for tablet c154e7d96dc64070bef1bccc85d9e51a (DEFAULT_TABLE table=default.Table_3ba14975b9434368bb1ad7ce335a7baa [id=9f4c126d1202439692b44a9bedfb2f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.820140 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ed4975796a547f1a8b84565712befde. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.820536 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c25912a79e82457c888b6d18a5643d32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.821445 2915 raft_consensus.cc:513] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.822556 908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abefc717e0eb41b18725ef0c73154fdd" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d"
I20250115 14:52:18.822726 908 raft_consensus.cc:3054] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:18.824232 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e94b77956d4b62862c52a739e44aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.824716 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c154e7d96dc64070bef1bccc85d9e51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.828392 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24e94b77956d4b62862c52a739e44aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.828507 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17189ed67174b9682bab32959b729e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.829031 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1423ccaf7dfc423ebba097f8f8585e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.831858 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be82f0894ca240e6954fc7af3c41ce23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.832384 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2c81edb81a474a9db7008ff12f2ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.835755 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea716ad60914d848a450eea082e696a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.835899 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abefc717e0eb41b18725ef0c73154fdd" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b"
W20250115 14:52:18.836658 984 leader_election.cc:343] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.838966 1162 tablet_bootstrap.cc:492] T b47502309761449dae3be07ea6363238 P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.839154 1162 ts_tablet_manager.cc:1397] T b47502309761449dae3be07ea6363238 P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.000s
I20250115 14:52:18.840333 1162 raft_consensus.cc:357] T b47502309761449dae3be07ea6363238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:18.840620 1162 raft_consensus.cc:383] T b47502309761449dae3be07ea6363238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.841039 1162 raft_consensus.cc:738] T b47502309761449dae3be07ea6363238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.841378 1162 consensus_queue.cc:260] T b47502309761449dae3be07ea6363238 P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:18.841634 2915 leader_election.cc:290] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Requested vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:18.842064 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ea716ad60914d848a450eea082e696a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.842240 1162 ts_tablet_manager.cc:1428] T b47502309761449dae3be07ea6363238 P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250115 14:52:18.842957 1166 ts_tablet_manager.cc:1918] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.843044 1166 log.cc:1198] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/a300a76b8f7942dea648a20cbb3d5368
I20250115 14:52:18.843294 1166 ts_tablet_manager.cc:1939] T a300a76b8f7942dea648a20cbb3d5368 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:18.844130 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet a300a76b8f7942dea648a20cbb3d5368 (table default.Table_0034e4c667b44bbb83a5a93e321e5113 [id=f6bae7686ebc45488f5eb27a554d28a3]) successfully deleted
I20250115 14:52:18.849392 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet 5882252383ba46aba0ec053fab1fd383 (table default.Table_049f35b1737c479bbff44ef1afe050e2 [id=1532f36247a340cbba3c9b792d500c04]). Expected version 2 got 0
I20250115 14:52:18.849594 451 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet fab76e0427f14da48bf101db57f0457c (table default.Table_4f9bd44ad3304deab5705c87466189da [id=61e995e8b453479992b8acbb14a05fb5]). Expected version 1 got 0
I20250115 14:52:18.850914 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_116236f57ba8476296d8aca8c9a7b470"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:18.851413 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_116236f57ba8476296d8aca8c9a7b470 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.853011 1164 tablet_bootstrap.cc:492] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:18.853148 1164 ts_tablet_manager.cc:1397] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.111s user 0.002s sys 0.002s
I20250115 14:52:18.854421 1027 tablet_service.cc:1467] Processing CreateTablet for tablet c154e7d96dc64070bef1bccc85d9e51a (DEFAULT_TABLE table=default.Table_3ba14975b9434368bb1ad7ce335a7baa [id=9f4c126d1202439692b44a9bedfb2f4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.854761 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c154e7d96dc64070bef1bccc85d9e51a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.855055 1167 ts_tablet_manager.cc:1918] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.855152 1167 log.cc:1198] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/0cae4c9228c640e495507e8f7ad9f544
I20250115 14:52:18.855401 1167 ts_tablet_manager.cc:1939] T 0cae4c9228c640e495507e8f7ad9f544 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:18.856065 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1391) took 1460 ms. Trace:
I20250115 14:52:18.856142 1167 rpcz_store.cc:276] 0115 14:52:17.395623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.399217 (+ 3594us) service_pool.cc:225] Handling call
0115 14:52:18.764163 (+1364946us) ts_tablet_manager.cc:1145] Deleting tablet 0cae4c9228c640e495507e8f7ad9f544
0115 14:52:18.765132 (+ 969us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.802314 (+ 37182us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.854952 (+ 52638us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.856046 (+ 1094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1364746}
I20250115 14:52:18.856451 1167 tablet_replica.cc:331] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:18.856626 1167 raft_consensus.cc:2238] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.856745 1167 raft_consensus.cc:2267] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.857193 1167 ts_tablet_manager.cc:1905] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.858279 1016 tablet_service.cc:1514] Processing DeleteTablet for tablet 87026b6faf75437eac21dedab1f41ed4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:16 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.858461 1023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3be094dcd3cb4f9d9259d354300c1600 with delete_type TABLET_DATA_DELETED (replaced by tablet b3daf6416ad847aeaa83087cc81c9239) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.865399 746 tablet_service.cc:1467] Processing CreateTablet for tablet ba1ff5041f6d4825a3d7380c792c3ba2 (DEFAULT_TABLE table=default.Table_cc2f5585e4594d849b5c784d7e6882cc [id=7e7b82d60ca647f380d01e7e9ac9c48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.865787 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba1ff5041f6d4825a3d7380c792c3ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.868772 1162 tablet_bootstrap.cc:492] T 2c6ac691e13a4d75be8a42a5d17d9eed P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.869228 1023 tablet_service.cc:1467] Processing CreateTablet for tablet a17189ed67174b9682bab32959b729e3 (DEFAULT_TABLE table=default.Table_7052ed5e2bf04d48b7d75dd702094b65 [id=bedc93977f754793b6eeb68aec944c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.869556 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17189ed67174b9682bab32959b729e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.870311 1162 tablet_bootstrap.cc:654] T 2c6ac691e13a4d75be8a42a5d17d9eed P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.853839 1164 raft_consensus.cc:357] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.871488 1164 raft_consensus.cc:383] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.871551 1164 raft_consensus.cc:738] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.871753 1164 consensus_queue.cc:260] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.873518 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 0cae4c9228c640e495507e8f7ad9f544 (table default.Table_dae457edf44a40a69a40030ae24a26ff [id=0dd965b7347c439d9abcc56446730332]) successfully deleted
I20250115 14:52:18.874676 884 tablet_service.cc:1467] Processing CreateTablet for tablet a17189ed67174b9682bab32959b729e3 (DEFAULT_TABLE table=default.Table_7052ed5e2bf04d48b7d75dd702094b65 [id=bedc93977f754793b6eeb68aec944c39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.875037 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a17189ed67174b9682bab32959b729e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.881848 746 tablet_service.cc:1467] Processing CreateTablet for tablet ffefb2f147144c4887f1fbdf4aca3a80 (DEFAULT_TABLE table=default.Table_9c6fe96ae2fd4ef790e39ed73291a0bb [id=5978ec04eeae40dd9bfe86d51c320357]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.882299 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffefb2f147144c4887f1fbdf4aca3a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.888165 1164 ts_tablet_manager.cc:1428] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20250115 14:52:18.889554 748 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6d2c9f516b4ebb89e2db3bb826f9e2 (DEFAULT_TABLE table=default.Table_b3cca34664f749cd9a8646474858530b [id=2d9d4a9c2be9415fb601a6f24b83ffd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.889926 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6d2c9f516b4ebb89e2db3bb826f9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.889818 1751 ts_tablet_manager.cc:1918] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.890192 1751 log.cc:1198] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/6203efb862ea4962a6d09f3e4306a92f
I20250115 14:52:18.890981 1751 ts_tablet_manager.cc:1939] T 6203efb862ea4962a6d09f3e4306a92f P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:18.891396 1016 tablet_service.cc:1467] Processing CreateTablet for tablet ffefb2f147144c4887f1fbdf4aca3a80 (DEFAULT_TABLE table=default.Table_9c6fe96ae2fd4ef790e39ed73291a0bb [id=5978ec04eeae40dd9bfe86d51c320357]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.897795 884 tablet_service.cc:1467] Processing CreateTablet for tablet ffefb2f147144c4887f1fbdf4aca3a80 (DEFAULT_TABLE table=default.Table_9c6fe96ae2fd4ef790e39ed73291a0bb [id=5978ec04eeae40dd9bfe86d51c320357]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.898260 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffefb2f147144c4887f1fbdf4aca3a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.899082 455 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 1218f0bdb1e14bd78d76c6a06805f38c
I20250115 14:52:18.899452 455 catalog_manager.cc:6596] ReplaceTablet: tablet 1218f0bdb1e14bd78d76c6a06805f38c deleted and replaced by tablet 2bde37806a1c4362831db07149adf433
I20250115 14:52:18.901605 455 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet b47502309761449dae3be07ea6363238 (table default.Table_57f588c93610423484d30d343513c135 [id=974866671a014ab6b7208c7b5824e8a3]). Expected version 1 got 0
I20250115 14:52:18.903712 1095 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:52:18.904834 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 36de948083a946eba1a96f276e4f4f91 with delete_type TABLET_DATA_DELETED (replaced by tablet c8e33ebec8b64f2398a7f1003b15cee7) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.905416 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 1218f0bdb1e14bd78d76c6a06805f38c with delete_type TABLET_DATA_DELETED (replaced by tablet 2bde37806a1c4362831db07149adf433) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.906419 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 14312c05485c4ebdaaddd0fcfd53e367
I20250115 14:52:18.907035 450 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_633028cf8b914cac8836453b1e242fdf" } modify_external_catalogs: true
I20250115 14:52:18.907193 450 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_633028cf8b914cac8836453b1e242fdf" } modify_external_catalogs: true
I20250115 14:52:18.908435 446 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 38e1b2ec1cb84b2f8b2d866f9f30296c
I20250115 14:52:18.908839 446 catalog_manager.cc:6596] ReplaceTablet: tablet 38e1b2ec1cb84b2f8b2d866f9f30296c deleted and replaced by tablet adf11bb6edda470f984f8560d8085510
I20250115 14:52:18.909093 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_cd7cf126bef040e7aac0c1b681b4370d" } new_table_name: "default.Table_c81bfff914bd43b5a2e43257fe91e30d" modify_external_catalogs: true
I20250115 14:52:18.909483 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c with delete_type TABLET_DATA_DELETED (replaced by tablet adf11bb6edda470f984f8560d8085510) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.909924 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 14312c05485c4ebdaaddd0fcfd53e367 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:18.910364 1164 tablet_bootstrap.cc:492] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:18.910492 453 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_65c87128c01649dea82c71db78a50391" } modify_external_catalogs: true
I20250115 14:52:18.910614 453 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_65c87128c01649dea82c71db78a50391" } modify_external_catalogs: true
I20250115 14:52:18.904213 455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4f0c9f99860448969e4dc7f8e1373b4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:18.912091 748 tablet_service.cc:1467] Processing CreateTablet for tablet 866454761ddb4455b7deb0231d413bc6 (DEFAULT_TABLE table=default.Table_494518ca300744e9821f893256ddf6a9 [id=8502b5fe264f4fb5a89bd4a91949c90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.912634 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866454761ddb4455b7deb0231d413bc6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250115 14:52:18.913009 455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f0c9f99860448969e4dc7f8e1373b4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:18.915325 1020 tablet_service.cc:1514] Processing DeleteTablet for tablet be8e12ccb8d14c1c9258dc3f81d1527d with delete_type TABLET_DATA_DELETED (replaced by tablet eba5819df2944a84a636c6934b18510d) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.915480 1029 tablet_service.cc:1514] Processing DeleteTablet for tablet 1218f0bdb1e14bd78d76c6a06805f38c with delete_type TABLET_DATA_DELETED (replaced by tablet 2bde37806a1c4362831db07149adf433) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.915625 1026 tablet_service.cc:1514] Processing DeleteTablet for tablet 14312c05485c4ebdaaddd0fcfd53e367 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.915827 1015 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c with delete_type TABLET_DATA_DELETED (replaced by tablet adf11bb6edda470f984f8560d8085510) from {username='slave'} at 127.0.0.1:43552
I20250115 14:52:18.916381 1015 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6d2c9f516b4ebb89e2db3bb826f9e2 (DEFAULT_TABLE table=default.Table_b3cca34664f749cd9a8646474858530b [id=2d9d4a9c2be9415fb601a6f24b83ffd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.917457 883 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c with delete_type TABLET_DATA_DELETED (replaced by tablet adf11bb6edda470f984f8560d8085510) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.919242 1013 tablet_service.cc:1467] Processing CreateTablet for tablet ba1ff5041f6d4825a3d7380c792c3ba2 (DEFAULT_TABLE table=default.Table_cc2f5585e4594d849b5c784d7e6882cc [id=7e7b82d60ca647f380d01e7e9ac9c48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.920410 1166 tablet_replica.cc:331] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:18.920626 1166 raft_consensus.cc:2238] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.920779 1166 raft_consensus.cc:2267] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.921546 1166 ts_tablet_manager.cc:1905] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.923038 448 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_c3dbb94e26f1439ba0bb576d883ec848" } new_table_name: "default.Table_5789433960f64cf5ad9da699c155b573" modify_external_catalogs: true
I20250115 14:52:18.925545 2549 raft_consensus.cc:491] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:18.925685 2549 raft_consensus.cc:513] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:18.926307 2549 leader_election.cc:290] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:18.927004 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3280e8ba6db4446aa3e3bfe2836abfa" candidate_uuid: "73c86158cc2641f182ea50a06634592d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
I20250115 14:52:18.927920 750 tablet_service.cc:1467] Processing CreateTablet for tablet 3390b801f8d44607af116e05983494c0 (DEFAULT_TABLE table=default.Table_7d66aaaa255c4c1a9672e2dd8b26a18e [id=c9a8dbeb02464c09badc44391d471fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.927906 1026 tablet_service.cc:1467] Processing CreateTablet for tablet 866454761ddb4455b7deb0231d413bc6 (DEFAULT_TABLE table=default.Table_494518ca300744e9821f893256ddf6a9 [id=8502b5fe264f4fb5a89bd4a91949c90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.928267 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3390b801f8d44607af116e05983494c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.928542 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5564827fdd41d69d44f2fe04a3f935 (DEFAULT_TABLE table=default.Table_2988e8cd3886441db242a362b06544c7 [id=e960b38b7ee44dea9fe699e47c861932]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:18.928982 848 leader_election.cc:343] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:18.929730 883 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5564827fdd41d69d44f2fe04a3f935 (DEFAULT_TABLE table=default.Table_2988e8cd3886441db242a362b06544c7 [id=e960b38b7ee44dea9fe699e47c861932]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.930064 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5564827fdd41d69d44f2fe04a3f935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.931196 1162 tablet_bootstrap.cc:492] T 2c6ac691e13a4d75be8a42a5d17d9eed P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:18.931385 1162 ts_tablet_manager.cc:1397] T 2c6ac691e13a4d75be8a42a5d17d9eed P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.006s
I20250115 14:52:18.932518 1162 raft_consensus.cc:357] T 2c6ac691e13a4d75be8a42a5d17d9eed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:18.932783 1162 raft_consensus.cc:383] T 2c6ac691e13a4d75be8a42a5d17d9eed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:18.932952 1162 raft_consensus.cc:738] T 2c6ac691e13a4d75be8a42a5d17d9eed P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:18.933267 1162 consensus_queue.cc:260] T 2c6ac691e13a4d75be8a42a5d17d9eed P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:18.934110 1162 ts_tablet_manager.cc:1428] T 2c6ac691e13a4d75be8a42a5d17d9eed P c238c037a1c74725a001e2b4a1ff4304: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250115 14:52:18.935639 750 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd702994525433db2fbbee1a423a52f (DEFAULT_TABLE table=default.Table_5dc7a5be3fe34c81892c07ff7348cdf2 [id=e5cc98bd1f9c4b0ba62cb1f559cf81f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.936038 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cd702994525433db2fbbee1a423a52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.937532 1166 ts_tablet_manager.cc:1918] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.937631 1166 log.cc:1198] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-0/wal/wals/36de948083a946eba1a96f276e4f4f91
I20250115 14:52:18.937886 1166 ts_tablet_manager.cc:1939] T 36de948083a946eba1a96f276e4f4f91 P d2d7ecece2ab49e1be7079216517d13b: Deleting consensus metadata
I20250115 14:52:18.938084 880 tablet_service.cc:1467] Processing CreateTablet for tablet 4a6d2c9f516b4ebb89e2db3bb826f9e2 (DEFAULT_TABLE table=default.Table_b3cca34664f749cd9a8646474858530b [id=2d9d4a9c2be9415fb601a6f24b83ffd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.938478 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6d2c9f516b4ebb89e2db3bb826f9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.938802 441 catalog_manager.cc:4872] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): tablet 36de948083a946eba1a96f276e4f4f91 (table default.Table_89373c553f9c407a9e205bfe0d1acf99 [id=26c53728ab22430b95ca322543118a43]) successfully deleted
I20250115 14:52:18.943588 891 tablet_service.cc:1514] Processing DeleteTablet for tablet 14312c05485c4ebdaaddd0fcfd53e367 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.944518 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 1218f0bdb1e14bd78d76c6a06805f38c with delete_type TABLET_DATA_DELETED (replaced by tablet 2bde37806a1c4362831db07149adf433) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.945214 1168 tablet_bootstrap.cc:654] T 1218f0bdb1e14bd78d76c6a06805f38c P 73c86158cc2641f182ea50a06634592d: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.945971 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a6d2c9f516b4ebb89e2db3bb826f9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.946439 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1411) took 1383 ms. Trace:
I20250115 14:52:18.946511 1751 rpcz_store.cc:276] 0115 14:52:17.563205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.563247 (+ 42us) service_pool.cc:225] Handling call
0115 14:52:18.778151 (+1214904us) ts_tablet_manager.cc:1145] Deleting tablet 6203efb862ea4962a6d09f3e4306a92f
0115 14:52:18.778160 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.866141 (+ 87981us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.889550 (+ 23409us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.945952 (+ 56402us) spinlock_profiling.cc:243] Waited 51 ms on lock 0x61800000fd18. stack: 00007f9adcf568a6 00007f9adcf56660 00007f9adbf3b48d 000000000096d544 000000000096d50d 000000000096ceda 00007f9adcf3f1ad 00007f9adcf4021e 00007f9af0371c7d 00007f9af02edd43 00007f9aec5913d3 00007f9aec5c3634 00007f9aec5a4740 00007f9aec5a44cd 0000000000a71ecc 00007f9adcf9d434
0115 14:52:18.946418 (+ 466us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":117271424,"tablet-delete.queue_time_us":1214772}
I20250115 14:52:18.947455 1751 tablet_replica.cc:331] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:18.947626 1751 raft_consensus.cc:2238] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:18.947769 1751 raft_consensus.cc:2267] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:18.948240 1751 ts_tablet_manager.cc:1905] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:18.948783 748 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5564827fdd41d69d44f2fe04a3f935 (DEFAULT_TABLE table=default.Table_2988e8cd3886441db242a362b06544c7 [id=e960b38b7ee44dea9fe699e47c861932]), partition=RANGE (key) PARTITION UNBOUNDED
W20250115 14:52:18.948957 847 leader_election.cc:343] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): Not found: Tablet not found: f3280e8ba6db4446aa3e3bfe2836abfa
I20250115 14:52:18.949069 847 leader_election.cc:304] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c86158cc2641f182ea50a06634592d; no voters: c238c037a1c74725a001e2b4a1ff4304, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:18.949255 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5564827fdd41d69d44f2fe04a3f935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.949708 2748 raft_consensus.cc:2743] T f3280e8ba6db4446aa3e3bfe2836abfa P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:18.952584 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866454761ddb4455b7deb0231d413bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.952620 893 tablet_service.cc:1467] Processing CreateTablet for tablet ba1ff5041f6d4825a3d7380c792c3ba2 (DEFAULT_TABLE table=default.Table_cc2f5585e4594d849b5c784d7e6882cc [id=7e7b82d60ca647f380d01e7e9ac9c48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.952800 742 tablet_service.cc:1467] Processing CreateTablet for tablet 74473f29e8164a3097922976cb8a8355 (DEFAULT_TABLE table=default.Table_62ac146a58da4b3b8ec38b3301c60339 [id=a1106aff96df4570ae9dd50ab14ce88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.952957 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba1ff5041f6d4825a3d7380c792c3ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.953114 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74473f29e8164a3097922976cb8a8355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.956945 1162 tablet_bootstrap.cc:492] T 05bfcbb5c8c84cc1adf2f8c52d2d2daa P c238c037a1c74725a001e2b4a1ff4304: Bootstrap starting.
I20250115 14:52:18.959074 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5564827fdd41d69d44f2fe04a3f935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.960654 747 tablet_service.cc:1467] Processing CreateTablet for tablet 14312c05485c4ebdaaddd0fcfd53e367 (DEFAULT_TABLE table=default.Table_213f670682f841518bdfae30c8c19d67 [id=4895ec4331804a8cb57d3c3b244ce277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.960901 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet 6203efb862ea4962a6d09f3e4306a92f (table default.Table_0636a1492a2542d99c65b5d1436dd981 [id=530bcb9252fa421b8df0f5932117a36c]) successfully deleted
I20250115 14:52:18.960986 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14312c05485c4ebdaaddd0fcfd53e367. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.962030 1014 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd702994525433db2fbbee1a423a52f (DEFAULT_TABLE table=default.Table_5dc7a5be3fe34c81892c07ff7348cdf2 [id=e5cc98bd1f9c4b0ba62cb1f559cf81f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.962330 1014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cd702994525433db2fbbee1a423a52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.964418 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffefb2f147144c4887f1fbdf4aca3a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.964574 749 tablet_service.cc:1467] Processing CreateTablet for tablet 4d01b740817b4537aeaaa5a35901ad40 (DEFAULT_TABLE table=default.Table_315961ed120f42f18e32f4612f7d30ee [id=9475807b870c49c09110f5e168a75c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.964797 742 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddb8db5a98f4ca285100f561aa8e6ac (DEFAULT_TABLE table=default.Table_aabd2a7266164dffa263161d840451bf [id=e5a8ff4796b04e378422eb587f5d5702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.964891 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d01b740817b4537aeaaa5a35901ad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.964941 1017 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddb8db5a98f4ca285100f561aa8e6ac (DEFAULT_TABLE table=default.Table_aabd2a7266164dffa263161d840451bf [id=e5a8ff4796b04e378422eb587f5d5702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.965087 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ddb8db5a98f4ca285100f561aa8e6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.965221 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ddb8db5a98f4ca285100f561aa8e6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.968648 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 14312c05485c4ebdaaddd0fcfd53e367 (DEFAULT_TABLE table=default.Table_213f670682f841518bdfae30c8c19d67 [id=4895ec4331804a8cb57d3c3b244ce277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.968981 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14312c05485c4ebdaaddd0fcfd53e367. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.971153 1013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba1ff5041f6d4825a3d7380c792c3ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.972333 746 tablet_service.cc:1467] Processing CreateTablet for tablet d5ab5da102d041179f72255a9e48a594 (DEFAULT_TABLE table=default.Table_a9ce413c298446edb254872574b66f69 [id=2b1ce956b036453cbb5bf8e860f5e4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.972733 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ab5da102d041179f72255a9e48a594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.977223 1029 tablet_service.cc:1467] Processing CreateTablet for tablet 74473f29e8164a3097922976cb8a8355 (DEFAULT_TABLE table=default.Table_62ac146a58da4b3b8ec38b3301c60339 [id=a1106aff96df4570ae9dd50ab14ce88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.980087 1012 tablet_service.cc:1467] Processing CreateTablet for tablet d5ab5da102d041179f72255a9e48a594 (DEFAULT_TABLE table=default.Table_a9ce413c298446edb254872574b66f69 [id=2b1ce956b036453cbb5bf8e860f5e4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.980424 884 tablet_service.cc:1514] Processing DeleteTablet for tablet ed777d7496514ec5986a7e6f53d532b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:18.980489 889 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f2bc62621d402cacfcbe2eb74173f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:09 UTC) from {username='slave'} at 127.0.0.1:56252
W20250115 14:52:18.980494 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 1218f0bdb1e14bd78d76c6a06805f38c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1218f0bdb1e14bd78d76c6a06805f38c already in progress: creating tablet
I20250115 14:52:18.981004 889 tablet_service.cc:1467] Processing CreateTablet for tablet 3390b801f8d44607af116e05983494c0 (DEFAULT_TABLE table=default.Table_7d66aaaa255c4c1a9672e2dd8b26a18e [id=c9a8dbeb02464c09badc44391d471fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.981338 892 tablet_service.cc:1467] Processing CreateTablet for tablet 9ddb8db5a98f4ca285100f561aa8e6ac (DEFAULT_TABLE table=default.Table_aabd2a7266164dffa263161d840451bf [id=e5a8ff4796b04e378422eb587f5d5702]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.981489 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3390b801f8d44607af116e05983494c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.981709 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ddb8db5a98f4ca285100f561aa8e6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.982512 1751 ts_tablet_manager.cc:1918] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:18.982661 1751 log.cc:1198] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/ad988e25800f4af9a069b5a80582f749
I20250115 14:52:18.983294 1751 ts_tablet_manager.cc:1939] T ad988e25800f4af9a069b5a80582f749 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
W20250115 14:52:18.985190 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 38e1b2ec1cb84b2f8b2d866f9f30296c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38e1b2ec1cb84b2f8b2d866f9f30296c already in progress: creating tablet
W20250115 14:52:18.985785 441 catalog_manager.cc:4852] TS d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): delete failed for tablet 14312c05485c4ebdaaddd0fcfd53e367 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14312c05485c4ebdaaddd0fcfd53e367 already in progress: creating tablet
I20250115 14:52:18.987066 1167 ts_tablet_manager.cc:1918] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:18.989261 1028 tablet_service.cc:1467] Processing CreateTablet for tablet 3390b801f8d44607af116e05983494c0 (DEFAULT_TABLE table=default.Table_7d66aaaa255c4c1a9672e2dd8b26a18e [id=c9a8dbeb02464c09badc44391d471fae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.990336 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74473f29e8164a3097922976cb8a8355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.993687 749 tablet_service.cc:1467] Processing CreateTablet for tablet c8327375fa5f4961b3a9f09d324d4276 (DEFAULT_TABLE table=default.Table_beee9856d4a747f48cfdeb296778ab88 [id=a829b99dc84641dbb2e4f7b2059f2ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:18.994063 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8327375fa5f4961b3a9f09d324d4276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.995579 1012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ab5da102d041179f72255a9e48a594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.000092 750 tablet_service.cc:1467] Processing CreateTablet for tablet b36a592d5d5749eebb8caa11b0d65e10 (DEFAULT_TABLE table=default.Table_a201de8c5f304323ac40334c8221a456 [id=a4a7e9e1b2514ba399a417727d2919d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.000519 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36a592d5d5749eebb8caa11b0d65e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.002163 742 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2c03a7362e4a22838a38b249b4ac38 (DEFAULT_TABLE table=default.Table_896e88aa54b34b63a9d7eb65d358e71b [id=21961d0c68f2420c8ee20b8197da6d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.002439 1027 tablet_service.cc:1467] Processing CreateTablet for tablet 4d01b740817b4537aeaaa5a35901ad40 (DEFAULT_TABLE table=default.Table_315961ed120f42f18e32f4612f7d30ee [id=9475807b870c49c09110f5e168a75c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.002604 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2c03a7362e4a22838a38b249b4ac38 (DEFAULT_TABLE table=default.Table_896e88aa54b34b63a9d7eb65d358e71b [id=21961d0c68f2420c8ee20b8197da6d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.002812 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2c03a7362e4a22838a38b249b4ac38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.002894 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d01b740817b4537aeaaa5a35901ad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.003248 1011 tablet_service.cc:1467] Processing CreateTablet for tablet b36a592d5d5749eebb8caa11b0d65e10 (DEFAULT_TABLE table=default.Table_a201de8c5f304323ac40334c8221a456 [id=a4a7e9e1b2514ba399a417727d2919d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.003640 1011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36a592d5d5749eebb8caa11b0d65e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.004864 1164 tablet_bootstrap.cc:654] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:19.006419 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2c03a7362e4a22838a38b249b4ac38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.007153 1164 tablet_bootstrap.cc:492] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:19.007280 1164 ts_tablet_manager.cc:1397] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.097s user 0.002s sys 0.002s
I20250115 14:52:19.007927 1164 raft_consensus.cc:357] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.008121 1164 raft_consensus.cc:383] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:19.008178 1164 raft_consensus.cc:738] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:19.008419 1164 consensus_queue.cc:260] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.009044 1164 ts_tablet_manager.cc:1428] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250115 14:52:19.009207 1167 log.cc:1198] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/18f23815196047628d562da7312b010f
I20250115 14:52:19.009497 1167 ts_tablet_manager.cc:1939] T 18f23815196047628d562da7312b010f P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:19.009795 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1392) took 1603 ms. Trace:
I20250115 14:52:18.988070 892 tablet_service.cc:1467] Processing CreateTablet for tablet 14312c05485c4ebdaaddd0fcfd53e367 (DEFAULT_TABLE table=default.Table_213f670682f841518bdfae30c8c19d67 [id=4895ec4331804a8cb57d3c3b244ce277]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.009876 1167 rpcz_store.cc:276] 0115 14:52:17.406786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.406837 (+ 51us) service_pool.cc:225] Handling call
0115 14:52:18.856409 (+1449572us) ts_tablet_manager.cc:1145] Deleting tablet 18f23815196047628d562da7312b010f
0115 14:52:18.856418 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.860003 (+ 3585us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.986981 (+126978us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.009777 (+ 22796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449368}
I20250115 14:52:19.010211 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14312c05485c4ebdaaddd0fcfd53e367. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.010264 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1404) took 1461 ms. Trace:
I20250115 14:52:19.010267 822 heartbeater.cc:346] Connected to a master server at 127.0.104.60:40227
I20250115 14:52:19.010331 1167 rpcz_store.cc:276] 0115 14:52:17.548837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.548883 (+ 46us) service_pool.cc:225] Handling call
0115 14:52:19.010149 (+1461266us) ts_tablet_manager.cc:1145] Deleting tablet bc06a77162a6430bbbda7a3ca2c4e9ab
0115 14:52:19.010156 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.010248 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1461090}
I20250115 14:52:19.010488 1168 tablet_bootstrap.cc:492] T 1218f0bdb1e14bd78d76c6a06805f38c P 73c86158cc2641f182ea50a06634592d: No bootstrap required, opened a new log
I20250115 14:52:19.010633 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1405) took 1449 ms. Trace:
I20250115 14:52:19.010638 1168 ts_tablet_manager.cc:1397] T 1218f0bdb1e14bd78d76c6a06805f38c P 73c86158cc2641f182ea50a06634592d: Time spent bootstrapping tablet: real 0.253s user 0.003s sys 0.001s
I20250115 14:52:19.010728 1167 rpcz_store.cc:276] 0115 14:52:17.561285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.561335 (+ 50us) service_pool.cc:225] Handling call
0115 14:52:19.010537 (+1449202us) ts_tablet_manager.cc:1145] Deleting tablet 417c5441bb7147a98b8ce0cd823a1855
0115 14:52:19.010545 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.010618 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1449039}
I20250115 14:52:19.011021 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1406) took 1449 ms. Trace:
I20250115 14:52:19.011088 1167 rpcz_store.cc:276] 0115 14:52:17.561974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.562099 (+ 125us) service_pool.cc:225] Handling call
0115 14:52:19.010928 (+1448829us) ts_tablet_manager.cc:1145] Deleting tablet df0d072330444439a6b9b87db099c8b1
0115 14:52:19.010936 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.011008 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448692}
I20250115 14:52:19.011336 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1407) took 1448 ms. Trace:
I20250115 14:52:19.011399 1167 rpcz_store.cc:276] 0115 14:52:17.562607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.562649 (+ 42us) service_pool.cc:225] Handling call
0115 14:52:19.011259 (+1448610us) ts_tablet_manager.cc:1145] Deleting tablet 87b28e83081a4f6e959c8b26c68b81ce
0115 14:52:19.011266 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.011325 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448489}
I20250115 14:52:19.011600 1167 tablet_replica.cc:331] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:19.011493 1168 raft_consensus.cc:357] T 1218f0bdb1e14bd78d76c6a06805f38c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.011734 1168 raft_consensus.cc:383] T 1218f0bdb1e14bd78d76c6a06805f38c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:19.011763 1167 raft_consensus.cc:2238] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 1 LEADER]: Raft consensus shutting down.
I20250115 14:52:19.011804 1168 raft_consensus.cc:738] T 1218f0bdb1e14bd78d76c6a06805f38c P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c86158cc2641f182ea50a06634592d, State: Initialized, Role: FOLLOWER
I20250115 14:52:19.012045 1167 raft_consensus.cc:2267] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:19.012148 1168 consensus_queue.cc:260] T 1218f0bdb1e14bd78d76c6a06805f38c P 73c86158cc2641f182ea50a06634592d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.012550 1164 tablet_bootstrap.cc:492] T 2560d42c15c64b4c9c0cfe95706b4d9e P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:19.014066 1164 tablet_bootstrap.cc:654] T 2560d42c15c64b4c9c0cfe95706b4d9e P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:18.988368 891 tablet_service.cc:1467] Processing CreateTablet for tablet d5ab5da102d041179f72255a9e48a594 (DEFAULT_TABLE table=default.Table_a9ce413c298446edb254872574b66f69 [id=2b1ce956b036453cbb5bf8e860f5e4ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.014645 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5ab5da102d041179f72255a9e48a594. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.988612 890 tablet_service.cc:1467] Processing CreateTablet for tablet 74473f29e8164a3097922976cb8a8355 (DEFAULT_TABLE table=default.Table_62ac146a58da4b3b8ec38b3301c60339 [id=a1106aff96df4570ae9dd50ab14ce88f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.017982 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74473f29e8164a3097922976cb8a8355. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:18.989754 882 tablet_service.cc:1467] Processing CreateTablet for tablet 866454761ddb4455b7deb0231d413bc6 (DEFAULT_TABLE table=default.Table_494518ca300744e9821f893256ddf6a9 [id=8502b5fe264f4fb5a89bd4a91949c90b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.020634 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866454761ddb4455b7deb0231d413bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.025447 877 tablet_service.cc:1467] Processing CreateTablet for tablet b36a592d5d5749eebb8caa11b0d65e10 (DEFAULT_TABLE table=default.Table_a201de8c5f304323ac40334c8221a456 [id=a4a7e9e1b2514ba399a417727d2919d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.025447 884 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2c03a7362e4a22838a38b249b4ac38 (DEFAULT_TABLE table=default.Table_896e88aa54b34b63a9d7eb65d358e71b [id=21961d0c68f2420c8ee20b8197da6d4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.025967 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36a592d5d5749eebb8caa11b0d65e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.026124 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d2c03a7362e4a22838a38b249b4ac38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.030855 1168 ts_tablet_manager.cc:1428] T 1218f0bdb1e14bd78d76c6a06805f38c P 73c86158cc2641f182ea50a06634592d: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20250115 14:52:19.031224 1168 tablet_bootstrap.cc:492] T 0f19f0a73ca24a37a5ef8e19b86ccc3c P 73c86158cc2641f182ea50a06634592d: Bootstrap starting.
I20250115 14:52:18.990549 908 raft_consensus.cc:2463] T abefc717e0eb41b18725ef0c73154fdd P 73c86158cc2641f182ea50a06634592d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c238c037a1c74725a001e2b4a1ff4304 in term 1.
I20250115 14:52:19.032192 452 catalog_manager.cc:5618] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351) does not have the latest schema for tablet a5e97279185541b992340c24ede01656 (table default.Table_a8391687956e4259a71808f267cb4ff0 [id=361ed20953ab4e79b1112ae76b95a423]). Expected version 3 got 0
I20250115 14:52:19.032377 983 leader_election.cc:304] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304; no voters: d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:19.032848 2915 raft_consensus.cc:2798] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 1 FOLLOWER]: Leader election won for term 1
I20250115 14:52:19.032963 2915 raft_consensus.cc:695] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [term 1 LEADER]: Becoming Leader. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Running, Role: LEADER
I20250115 14:52:19.033196 2915 consensus_queue.cc:237] T abefc717e0eb41b18725ef0c73154fdd P c238c037a1c74725a001e2b4a1ff4304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.035663 749 tablet_service.cc:1514] Processing DeleteTablet for tablet a48a0af9422f43cf86531665c0fa1aee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:04 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:19.036259 889 tablet_service.cc:1467] Processing CreateTablet for tablet 4d01b740817b4537aeaaa5a35901ad40 (DEFAULT_TABLE table=default.Table_315961ed120f42f18e32f4612f7d30ee [id=9475807b870c49c09110f5e168a75c96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.036616 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d01b740817b4537aeaaa5a35901ad40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.037492 452 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_4175560ad7c04cf59d79eab9bb95c166"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:19.038126 452 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4175560ad7c04cf59d79eab9bb95c166 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:19.047627 1027 tablet_service.cc:1467] Processing CreateTablet for tablet c8327375fa5f4961b3a9f09d324d4276 (DEFAULT_TABLE table=default.Table_beee9856d4a747f48cfdeb296778ab88 [id=a829b99dc84641dbb2e4f7b2059f2ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.048018 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8327375fa5f4961b3a9f09d324d4276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.050747 748 tablet_service.cc:1467] Processing CreateTablet for tablet d7be0783b0bd45c68e712d06e6f742d7 (DEFAULT_TABLE table=default.Table_54eea988fc86453db7e4a99597a5e552 [id=b202a4fc10484f7e8c06c6e2710acc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.051277 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7be0783b0bd45c68e712d06e6f742d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.052949 1029 tablet_service.cc:1467] Processing CreateTablet for tablet d7be0783b0bd45c68e712d06e6f742d7 (DEFAULT_TABLE table=default.Table_54eea988fc86453db7e4a99597a5e552 [id=b202a4fc10484f7e8c06c6e2710acc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.053417 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7be0783b0bd45c68e712d06e6f742d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.056365 1167 ts_tablet_manager.cc:1905] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:19.057307 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1412) took 1492 ms. Trace:
I20250115 14:52:19.057405 1751 rpcz_store.cc:276] 0115 14:52:17.564572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.564613 (+ 41us) service_pool.cc:225] Handling call
0115 14:52:18.946781 (+1382168us) ts_tablet_manager.cc:1145] Deleting tablet ad988e25800f4af9a069b5a80582f749
0115 14:52:18.947399 (+ 618us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:18.951023 (+ 3624us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:18.982383 (+ 31360us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.057287 (+ 74904us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1382027}
I20250115 14:52:19.057891 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3390b801f8d44607af116e05983494c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.058617 1166 tablet_replica.cc:331] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b: stopping tablet replica
I20250115 14:52:19.058812 1166 raft_consensus.cc:2238] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:19.058964 1166 raft_consensus.cc:2267] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:19.059430 884 tablet_service.cc:1467] Processing CreateTablet for tablet d7be0783b0bd45c68e712d06e6f742d7 (DEFAULT_TABLE table=default.Table_54eea988fc86453db7e4a99597a5e552 [id=b202a4fc10484f7e8c06c6e2710acc6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.059816 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7be0783b0bd45c68e712d06e6f742d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.061589 1164 tablet_bootstrap.cc:492] T 2560d42c15c64b4c9c0cfe95706b4d9e P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:19.061800 1164 ts_tablet_manager.cc:1397] T 2560d42c15c64b4c9c0cfe95706b4d9e P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20250115 14:52:19.063308 1164 raft_consensus.cc:357] T 2560d42c15c64b4c9c0cfe95706b4d9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.063644 1164 raft_consensus.cc:383] T 2560d42c15c64b4c9c0cfe95706b4d9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:19.063755 1164 raft_consensus.cc:738] T 2560d42c15c64b4c9c0cfe95706b4d9e P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:19.064092 1164 consensus_queue.cc:260] T 2560d42c15c64b4c9c0cfe95706b4d9e P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.068459 1164 ts_tablet_manager.cc:1428] T 2560d42c15c64b4c9c0cfe95706b4d9e P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250115 14:52:19.069000 889 tablet_service.cc:1467] Processing CreateTablet for tablet c8327375fa5f4961b3a9f09d324d4276 (DEFAULT_TABLE table=default.Table_beee9856d4a747f48cfdeb296778ab88 [id=a829b99dc84641dbb2e4f7b2059f2ff4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.069515 1164 tablet_bootstrap.cc:492] T ed777d7496514ec5986a7e6f53d532b1 P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:19.071854 2839 raft_consensus.cc:491] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:19.071992 2839 raft_consensus.cc:513] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.072690 2839 leader_election.cc:290] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:19.073341 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14836b97c3247b39ff9072c12e642e0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
W20250115 14:52:19.074100 983 leader_election.cc:343] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:19.073422 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8327375fa5f4961b3a9f09d324d4276. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.078505 1167 ts_tablet_manager.cc:1918] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250115 14:52:19.079602 1164 tablet_bootstrap.cc:654] T ed777d7496514ec5986a7e6f53d532b1 P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:19.079818 1166 ts_tablet_manager.cc:1905] T a48a0af9422f43cf86531665c0fa1aee P d2d7ecece2ab49e1be7079216517d13b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:19.081583 775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c14836b97c3247b39ff9072c12e642e0" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:19.082069 984 leader_election.cc:343] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:19.082201 984 leader_election.cc:304] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c238c037a1c74725a001e2b4a1ff4304; no voters: 73c86158cc2641f182ea50a06634592d, d2d7ecece2ab49e1be7079216517d13b
I20250115 14:52:19.082608 2839 raft_consensus.cc:2743] T c14836b97c3247b39ff9072c12e642e0 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:19.084100 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet 18f23815196047628d562da7312b010f (table default.Table_b053773cca094e6499791ad922f1fe10 [id=bd973eb84bcb426fa3e5963b7cb949dd]) successfully deleted
W20250115 14:52:19.084369 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet bc06a77162a6430bbbda7a3ca2c4e9ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc06a77162a6430bbbda7a3ca2c4e9ab already in progress: creating tablet
W20250115 14:52:19.084553 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 417c5441bb7147a98b8ce0cd823a1855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 417c5441bb7147a98b8ce0cd823a1855 already in progress: creating tablet
W20250115 14:52:19.084702 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet df0d072330444439a6b9b87db099c8b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df0d072330444439a6b9b87db099c8b1 already in progress: creating tablet
W20250115 14:52:19.084841 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 87b28e83081a4f6e959c8b26c68b81ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87b28e83081a4f6e959c8b26c68b81ce already in progress: creating tablet
I20250115 14:52:19.085846 440 catalog_manager.cc:4872] TS c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351): tablet ad988e25800f4af9a069b5a80582f749 (table default.Table_162eb94aa1674d21ae644021401a54e8 [id=e3887aa962874cfaa7ee89503ca1a49a]) successfully deleted
I20250115 14:52:19.090298 1162 tablet_bootstrap.cc:654] T 05bfcbb5c8c84cc1adf2f8c52d2d2daa P c238c037a1c74725a001e2b4a1ff4304: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:19.097429 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1416) took 1476 ms. Trace:
I20250115 14:52:19.097627 1751 rpcz_store.cc:276] 0115 14:52:17.621253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.621539 (+ 286us) service_pool.cc:225] Handling call
0115 14:52:19.097272 (+1475733us) ts_tablet_manager.cc:1145] Deleting tablet 9449e1d09c3148ee86074a57c8e22240
0115 14:52:19.097283 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.097405 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1475536}
I20250115 14:52:19.098011 1751 tablet_replica.cc:331] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:19.098248 1751 raft_consensus.cc:2238] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:19.098448 1751 raft_consensus.cc:2267] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:19.099023 1751 ts_tablet_manager.cc:1905] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:19.100862 3001 raft_consensus.cc:491] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:19.101014 3001 raft_consensus.cc:513] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.101619 3001 leader_election.cc:290] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187), c238c037a1c74725a001e2b4a1ff4304 (127.0.104.3:36351)
I20250115 14:52:19.106905 1043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04daad461f0b433ba635df87439b5bc1" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c238c037a1c74725a001e2b4a1ff4304" is_pre_election: true
I20250115 14:52:19.107614 447 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 90764f6eaaba46ee9cdc70672d9fe732
I20250115 14:52:19.109202 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 90764f6eaaba46ee9cdc70672d9fe732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:19.113201 454 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet 9c35882d2e5c464c9cfad496ba9d6af9
I20250115 14:52:19.113546 450 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet ce693a5c338f4e7394e0f5c323b8942b
I20250115 14:52:19.114226 447 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_36b868736e9e46528bddc87ba5998ff8" } modify_external_catalogs: true
I20250115 14:52:19.114226 450 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_a9f462afda4b453781827df3f7fd0aaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250115 14:52:19.114395 447 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_36b868736e9e46528bddc87ba5998ff8" } modify_external_catalogs: true
W20250115 14:52:19.114773 450 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9f462afda4b453781827df3f7fd0aaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250115 14:52:19.115804 446 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0375eea766464ac195ee97d447687dd6 consensus state has no leader
I20250115 14:52:19.116125 446 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_2d510cf7315e46bab6863551ac6bf816" } new_table_name: "default.Table_3421fb366eb24ee78918c5925bb6072d" modify_external_catalogs: true
I20250115 14:52:19.117393 454 catalog_manager.cc:6596] ReplaceTablet: tablet 9c35882d2e5c464c9cfad496ba9d6af9 deleted and replaced by tablet 1dcc43c6675c4af4bbea23f9863540c5
I20250115 14:52:19.117585 454 master_service.cc:946] ReplaceTablet: received request to replace tablet a9f2e5eadb54419a9071d64474073e34 from {username='slave'} at 127.0.0.1:47464
I20250115 14:52:19.120929 449 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_f58997a623b54f1892fe42ef782fc3af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:19.121551 449 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f58997a623b54f1892fe42ef782fc3af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:19.108425 893 tablet_service.cc:1467] Processing CreateTablet for tablet 3cd702994525433db2fbbee1a423a52f (DEFAULT_TABLE table=default.Table_5dc7a5be3fe34c81892c07ff7348cdf2 [id=e5cc98bd1f9c4b0ba62cb1f559cf81f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250115 14:52:19.125124 748 tablet_service.cc:1514] Processing DeleteTablet for tablet ce693a5c338f4e7394e0f5c323b8942b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:19.112515 911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04daad461f0b433ba635df87439b5bc1" candidate_uuid: "d2d7ecece2ab49e1be7079216517d13b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c86158cc2641f182ea50a06634592d" is_pre_election: true
I20250115 14:52:19.125849 710 leader_election.cc:304] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2d7ecece2ab49e1be7079216517d13b; no voters: 73c86158cc2641f182ea50a06634592d, c238c037a1c74725a001e2b4a1ff4304
I20250115 14:52:19.126251 3001 raft_consensus.cc:2743] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250115 14:52:19.126466 750 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c35882d2e5c464c9cfad496ba9d6af9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dcc43c6675c4af4bbea23f9863540c5) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:19.126715 1167 log.cc:1198] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ad988e25800f4af9a069b5a80582f749
I20250115 14:52:19.127221 1167 ts_tablet_manager.cc:1939] T ad988e25800f4af9a069b5a80582f749 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:19.127976 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1408) took 1550 ms. Trace:
I20250115 14:52:19.128579 882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c35882d2e5c464c9cfad496ba9d6af9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1dcc43c6675c4af4bbea23f9863540c5) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:19.128535 890 tablet_service.cc:1514] Processing DeleteTablet for tablet ce693a5c338f4e7394e0f5c323b8942b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:19.128526 1167 rpcz_store.cc:276] 0115 14:52:17.577069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.577131 (+ 62us) service_pool.cc:225] Handling call
0115 14:52:19.011567 (+1434436us) ts_tablet_manager.cc:1145] Deleting tablet ad988e25800f4af9a069b5a80582f749
0115 14:52:19.011574 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.068220 (+ 56646us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.078419 (+ 10199us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.127961 (+ 49542us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1434257}
I20250115 14:52:19.129576 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1411) took 1508 ms. Trace:
I20250115 14:52:19.130517 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet ad988e25800f4af9a069b5a80582f749 (table default.Table_162eb94aa1674d21ae644021401a54e8 [id=e3887aa962874cfaa7ee89503ca1a49a]) successfully deleted
I20250115 14:52:19.130919 1167 rpcz_store.cc:276] 0115 14:52:17.621501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.621549 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:19.129467 (+1507918us) ts_tablet_manager.cc:1145] Deleting tablet 9449e1d09c3148ee86074a57c8e22240
0115 14:52:19.129476 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.129561 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1507737}
I20250115 14:52:19.131716 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1413) took 1316 ms. Trace:
I20250115 14:52:19.131790 1167 rpcz_store.cc:276] 0115 14:52:17.815315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.815363 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:19.131610 (+1316247us) ts_tablet_manager.cc:1145] Deleting tablet ddb66753bb00448c8030b0e0ff03bb22
0115 14:52:19.131619 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.131700 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1316077}
W20250115 14:52:19.131863 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet 9449e1d09c3148ee86074a57c8e22240 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9449e1d09c3148ee86074a57c8e22240 already in progress: creating tablet
I20250115 14:52:19.132155 1167 tablet_replica.cc:331] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
W20250115 14:52:19.132174 440 catalog_manager.cc:4852] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): delete failed for tablet ddb66753bb00448c8030b0e0ff03bb22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddb66753bb00448c8030b0e0ff03bb22 already in progress: creating tablet
I20250115 14:52:19.132373 1167 raft_consensus.cc:2238] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:19.132525 1167 raft_consensus.cc:2267] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:19.133038 1167 ts_tablet_manager.cc:1905] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:19.133394 453 catalog_manager.cc:5813] T 00000000000000000000000000000000 P ef770dae727b42f5a9513b6f3d68b84b: Sending DeleteTablet for 3 replicas of tablet f869a6433f714f1999b41063fdd6b4ec
I20250115 14:52:19.134128 1162 tablet_bootstrap.cc:492] T 05bfcbb5c8c84cc1adf2f8c52d2d2daa P c238c037a1c74725a001e2b4a1ff4304: No bootstrap required, opened a new log
I20250115 14:52:19.134253 1162 ts_tablet_manager.cc:1397] T 05bfcbb5c8c84cc1adf2f8c52d2d2daa P c238c037a1c74725a001e2b4a1ff4304: Time spent bootstrapping tablet: real 0.177s user 0.002s sys 0.008s
I20250115 14:52:19.135267 1162 raft_consensus.cc:357] T 05bfcbb5c8c84cc1adf2f8c52d2d2daa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:19.135460 1162 raft_consensus.cc:383] T 05bfcbb5c8c84cc1adf2f8c52d2d2daa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:19.135507 1162 raft_consensus.cc:738] T 05bfcbb5c8c84cc1adf2f8c52d2d2daa P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c238c037a1c74725a001e2b4a1ff4304, State: Initialized, Role: FOLLOWER
I20250115 14:52:19.135699 1162 consensus_queue.cc:260] T 05bfcbb5c8c84cc1adf2f8c52d2d2daa P c238c037a1c74725a001e2b4a1ff4304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } }
I20250115 14:52:19.135807 453 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_e82073df06d449c3870cf6d7b8133825" } new_table_name: "default.Table_da4b08ef9eaa49a6a6b99b505aecde70" modify_external_catalogs: true
I20250115 14:52:19.130381 877 tablet_service.cc:1514] Processing DeleteTablet for tablet 90764f6eaaba46ee9cdc70672d9fe732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:19.129283 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cd702994525433db2fbbee1a423a52f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250115 14:52:19.141870 748 tablet_service.cc:1514] Processing DeleteTablet for tablet 2560d42c15c64b4c9c0cfe95706b4d9e with delete_type TABLET_DATA_DELETED (replaced by tablet f4536bb3d44b40d2825e1f4e59d8a148) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:19.142046 750 tablet_service.cc:1514] Processing DeleteTablet for tablet f869a6433f714f1999b41063fdd6b4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:46776
I20250115 14:52:19.143658 451 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47464:
name: "default.Table_49c67bb5712a43309d2f5927cc875ac0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250115 14:52:19.144160 451 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49c67bb5712a43309d2f5927cc875ac0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250115 14:52:19.145635 455 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47464:
table { table_name: "default.Table_5d9f72946dd846b98daa1df0c392cb72" } new_table_name: "default.Table_9eee3ba1cb2747b2ac6179adb2edd0de" modify_external_catalogs: true
I20250115 14:52:19.153373 775 raft_consensus.cc:3054] T 04daad461f0b433ba635df87439b5bc1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Advancing to term 1
I20250115 14:52:19.156734 877 tablet_service.cc:1514] Processing DeleteTablet for tablet f869a6433f714f1999b41063fdd6b4ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-15 14:52:18 UTC) from {username='slave'} at 127.0.0.1:56252
I20250115 14:52:19.157058 1164 tablet_bootstrap.cc:492] T ed777d7496514ec5986a7e6f53d532b1 P d2d7ecece2ab49e1be7079216517d13b: No bootstrap required, opened a new log
I20250115 14:52:19.157856 1164 ts_tablet_manager.cc:1397] T ed777d7496514ec5986a7e6f53d532b1 P d2d7ecece2ab49e1be7079216517d13b: Time spent bootstrapping tablet: real 0.089s user 0.013s sys 0.007s
I20250115 14:52:19.162659 1164 raft_consensus.cc:357] T ed777d7496514ec5986a7e6f53d532b1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.163874 1164 raft_consensus.cc:383] T ed777d7496514ec5986a7e6f53d532b1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250115 14:52:19.164274 1164 raft_consensus.cc:738] T ed777d7496514ec5986a7e6f53d532b1 P d2d7ecece2ab49e1be7079216517d13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d7ecece2ab49e1be7079216517d13b, State: Initialized, Role: FOLLOWER
I20250115 14:52:19.164623 3042 raft_consensus.cc:491] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250115 14:52:19.164762 3042 raft_consensus.cc:513] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.165433 3042 leader_election.cc:290] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127), 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187)
I20250115 14:52:19.165911 774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03eb225fad7459a955a3ec5e0002be1" candidate_uuid: "c238c037a1c74725a001e2b4a1ff4304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2d7ecece2ab49e1be7079216517d13b" is_pre_election: true
W20250115 14:52:19.166486 984 leader_election.cc:343] T d03eb225fad7459a955a3ec5e0002be1 P c238c037a1c74725a001e2b4a1ff4304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d2d7ecece2ab49e1be7079216517d13b (127.0.104.1:34127): Illegal state: must be running to vote when last-logged opid is not known
I20250115 14:52:19.165652 1164 consensus_queue.cc:260] T ed777d7496514ec5986a7e6f53d532b1 P d2d7ecece2ab49e1be7079216517d13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2d7ecece2ab49e1be7079216517d13b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34127 } } peers { permanent_uuid: "73c86158cc2641f182ea50a06634592d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44187 } } peers { permanent_uuid: "c238c037a1c74725a001e2b4a1ff4304" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36351 } }
I20250115 14:52:19.169270 1164 ts_tablet_manager.cc:1428] T ed777d7496514ec5986a7e6f53d532b1 P d2d7ecece2ab49e1be7079216517d13b: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20250115 14:52:19.170704 1751 ts_tablet_manager.cc:1918] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:19.170804 1751 log.cc:1198] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/11fddbd9ba14443a91865387e1d1c238
I20250115 14:52:19.171072 1751 ts_tablet_manager.cc:1939] T 11fddbd9ba14443a91865387e1d1c238 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:19.170976 1164 tablet_bootstrap.cc:492] T abefc717e0eb41b18725ef0c73154fdd P d2d7ecece2ab49e1be7079216517d13b: Bootstrap starting.
I20250115 14:52:19.171211 1167 ts_tablet_manager.cc:1918] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:19.171305 1167 log.cc:1198] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-1/wal/wals/ba33eb3f5a40450db5cf115fbe8a17e7
I20250115 14:52:19.171352 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1417) took 1518 ms. Trace:
I20250115 14:52:19.171417 1751 rpcz_store.cc:276] 0115 14:52:17.652507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.652559 (+ 52us) service_pool.cc:225] Handling call
0115 14:52:19.097969 (+1445410us) ts_tablet_manager.cc:1145] Deleting tablet 11fddbd9ba14443a91865387e1d1c238
0115 14:52:19.097978 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.127211 (+ 29233us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.170603 (+ 43392us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.171337 (+ 734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445237}
I20250115 14:52:19.171545 1167 ts_tablet_manager.cc:1939] T ba33eb3f5a40450db5cf115fbe8a17e7 P 73c86158cc2641f182ea50a06634592d: Deleting consensus metadata
I20250115 14:52:19.171706 1751 tablet_replica.cc:331] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:19.171793 1167 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56252 (request call id 1414) took 1325 ms. Trace:
I20250115 14:52:19.171881 1751 raft_consensus.cc:2238] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:19.171855 1167 rpcz_store.cc:276] 0115 14:52:17.846587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.846634 (+ 47us) service_pool.cc:225] Handling call
0115 14:52:19.132113 (+1285479us) ts_tablet_manager.cc:1145] Deleting tablet ba33eb3f5a40450db5cf115fbe8a17e7
0115 14:52:19.132122 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.155196 (+ 23074us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.171116 (+ 15920us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.171779 (+ 663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1285304}
I20250115 14:52:19.172009 1751 raft_consensus.cc:2267] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:19.172232 440 catalog_manager.cc:4872] TS 73c86158cc2641f182ea50a06634592d (127.0.104.2:44187): tablet ba33eb3f5a40450db5cf115fbe8a17e7 (table default.Table_ff33645127a443efb2b776d808942dba [id=f8c4ca50220a4dbf9b9480213d836e5f]) successfully deleted
I20250115 14:52:19.175812 1751 ts_tablet_manager.cc:1905] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:19.177763 1167 tablet_replica.cc:331] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d: stopping tablet replica
I20250115 14:52:19.177960 1167 raft_consensus.cc:2238] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:19.178128 1167 raft_consensus.cc:2267] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:19.180675 1164 tablet_bootstrap.cc:654] T abefc717e0eb41b18725ef0c73154fdd P d2d7ecece2ab49e1be7079216517d13b: Neither blocks nor log segments found. Creating new log.
I20250115 14:52:19.183861 1751 ts_tablet_manager.cc:1918] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:19.183969 1751 log.cc:1198] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/7d612b424bea408cb6691c7c200cf3a6
I20250115 14:52:19.184232 1751 ts_tablet_manager.cc:1939] T 7d612b424bea408cb6691c7c200cf3a6 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:19.184607 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1418) took 1441 ms. Trace:
I20250115 14:52:19.184698 1751 rpcz_store.cc:276] 0115 14:52:17.742834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.742882 (+ 48us) service_pool.cc:225] Handling call
0115 14:52:19.171658 (+1428776us) ts_tablet_manager.cc:1145] Deleting tablet 7d612b424bea408cb6691c7c200cf3a6
0115 14:52:19.171667 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.180567 (+ 8900us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.183760 (+ 3193us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.184588 (+ 828us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1428623}
I20250115 14:52:19.184931 1167 ts_tablet_manager.cc:1905] T 2560d42c15c64b4c9c0cfe95706b4d9e P 73c86158cc2641f182ea50a06634592d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:19.185029 1751 tablet_replica.cc:331] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: stopping tablet replica
I20250115 14:52:19.185204 1751 raft_consensus.cc:2238] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250115 14:52:19.185353 1751 raft_consensus.cc:2267] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250115 14:52:19.185881 1751 ts_tablet_manager.cc:1905] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250115 14:52:19.191920 1751 ts_tablet_manager.cc:1918] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250115 14:52:19.192026 1751 log.cc:1198] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: Deleting WAL directory at /tmp/dist-test-task__Z8ae/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736952707239278-416-0/minicluster-data/ts-2/wal/wals/d18a88c8f3ce4a24b7e1e13966760364
I20250115 14:52:19.192268 1751 ts_tablet_manager.cc:1939] T d18a88c8f3ce4a24b7e1e13966760364 P c238c037a1c74725a001e2b4a1ff4304: Deleting consensus metadata
I20250115 14:52:19.192584 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1419) took 1404 ms. Trace:
I20250115 14:52:19.192657 1751 rpcz_store.cc:276] 0115 14:52:17.788446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.788501 (+ 55us) service_pool.cc:225] Handling call
0115 14:52:19.184986 (+1396485us) ts_tablet_manager.cc:1145] Deleting tablet d18a88c8f3ce4a24b7e1e13966760364
0115 14:52:19.184996 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.189040 (+ 4044us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.191845 (+ 2805us) tablet_metadata.cc:641] Metadata flushed
0115 14:52:19.192569 (+ 724us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1396313}
I20250115 14:52:19.193024 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1420) took 1379 ms. Trace:
I20250115 14:52:19.193095 1751 rpcz_store.cc:276] 0115 14:52:17.813248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.813313 (+ 65us) service_pool.cc:225] Handling call
0115 14:52:19.192914 (+1379601us) ts_tablet_manager.cc:1145] Deleting tablet ddb66753bb00448c8030b0e0ff03bb22
0115 14:52:19.192922 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.193012 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1379404}
I20250115 14:52:19.193470 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1427) took 1254 ms. Trace:
I20250115 14:52:19.193539 1751 rpcz_store.cc:276] 0115 14:52:17.938630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.938850 (+ 220us) service_pool.cc:225] Handling call
0115 14:52:19.193378 (+1254528us) ts_tablet_manager.cc:1145] Deleting tablet 08d9f83287c6450cb30d0d02dc1c6238
0115 14:52:19.193387 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.193457 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254376}
I20250115 14:52:19.193820 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1428) took 1255 ms. Trace:
I20250115 14:52:19.193893 1751 rpcz_store.cc:276] 0115 14:52:17.938718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.939043 (+ 325us) service_pool.cc:225] Handling call
0115 14:52:19.193719 (+1254676us) ts_tablet_manager.cc:1145] Deleting tablet 10c9de0f178743f88e1334c5397e4fd3
0115 14:52:19.193728 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.193805 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254559}
I20250115 14:52:19.194159 1751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43552 (request call id 1429) took 1255 ms. Trace:
I20250115 14:52:19.194228 1751 rpcz_store.cc:276] 0115 14:52:17.938845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0115 14:52:17.939181 (+ 336us) service_pool.cc:225] Handling call
0115 14:52:19.194076 (+1254895us) ts_tablet_manager.cc:1145] Deleting tablet ae1a9330d9fd4cd5b629af7e1f44ab0e
0115 14:52:19.194084 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0115 14:52:19.194147 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us"